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Foreword 



Although written by a professional evaluator, this monograph is not about evaluation. It 
is about delivering better programs - programs that are not only more plausible but more 
impactful and robust as well. It is based on many years of experience in attempting to 
evaluate educational programs, especially one's that are non-formal and involve learning- 
by-doing or, what is often called experiential learning. The Cooperative Extension System 
is especially prominent and perhaps the foremost organization in the world, in providing 
such programs. 

This monograph is meant to serve as a conceptual guide for those facilitating or 
conducting life-cycle program management workshops focused on any of the life-cycle 
stages of: problem finding; program design; program development; program 
implementation; program maintenance and improvement; and, program redirection. It is 
hoped that these chapters will provide a beginning framework which will be amended and 
emended from the experiences gained in the workshops as well as from their later results 
- hence the approach is called "heuristic". 

It is anticipated that such workshops will focus on topics that involve a multiplicity of 
actors for whom a concerted, coordinated effort is required or desired. Topics that cut 
across organizational, disciplinary &/or geographic boundaries and require some special 
effort are especially suited to such an approach. There is a heavy reliance on the use of 
visual materials since graphic presentations seem to readily facilitate "working in groups" 
while reliance on printed matter can be an impediment. 

Although this monograph is not intended to foster an employment program for 
professional evaluators they are particularly well suited to serve as the facilitators of life- 
cycle workshops. They combine a happy blend of research skills with a practical, user 
oriented approach reinforced by a high degree of interpersonal skill. In short, they are the 
natural heirs to such an approach (albeit not the only heirs). Perhaps they will even 
pursue a set of standards for programs similar to those developed for evaluations some 
yearn ago (Stufflebeam, 1981). Certainly, Mueller (1991) and Smith (1991) have taken 
major steps in that direction, at least for Extension programs. 

The names of my colleagues to whom I am indebted for their help in these efforts is 
myriad in number and geographic locale, ranging from Maine to the Marianas Islands. In 
my work I have found them to be a walking "treasure trove" of "know-how" and "can-do- 
ism's". All would be richer by far if much of that experience-based ingenuity could be 
systematized and documented. It is hoped that this monograph takes a small step in that 
direction. Special notes of thanks are due to: John S. Bottum, Deputy Administrator for 
Planning, Development and Evaluation for his continuing support of this work; "Midge" 
Smith for her pioneering work in adapting Evaluability Assessment (EA) techniques to 
Extension programming; to Joe Wholey for starting EA in the first place (and who may 
be scratching his head in wonderment as to how it could have come to this); to a network 
of colleagues who helped me carry out a "whole bunch" of EA's while along the way 
adapting these techniques to design future programs rather than attempting to "resurrect" 
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old ones (Charles Clark, Debbie Killam, Michael Lambur, John Michael, Marjorie Mortvedt, 
Maria Russell, Satish Verma and Randall Workman) and to my colleague Leon Hunter 
for getting me started with a graphics package - it has "made my day" many times over. 
I am especially indebted to the Louisiana 4-H Youth Development Design Team for 
orienting my thinking about local agents as coordinators of program delivery teams and 
to the North Carolina A&T Adolescent Pregnancy and Parenting Team for allowing me 
to work with them as they carried their design into the developmental phase - an effort 
from which I continue to learn much. The continuing work of Mike Lambur and Judy 
Burtner on devising means to identify programs for elimination will be especially 
instructive for all, as it nears fruition. I touched on some of their work in Chapter 13 for 
it is both excellent and begins to fill a void in our collective knowledge base about ending 
programs. Finally a special note of thanks needs to be given to my Secretary Rosa L. 
Monroe who has labored for years over many incarnations of this work (of which this 
latest will likely also be its last). At times she was ably assisted by Mia Johnson. 

The illustrative program used as an example of a design which is then carried into 
development and implementation, is actually based on a synthesis of most of the program 
design and evaluability assessment work done in Extension to date. It is believed to have 
wide applicability for Extension programming. There is a great deal of developmental work 
that needs to be done yet for the different life-cycle workshops. Most is known by far, 
about procedures for the program design workshops, as is reflected in the amount ■•: 
space devoted to the topic. Hopefully, similar experiences will accrue for the other cycles. 

Finally, a note is in order about the Life Cycle Guidance Team. This theoretical construct 
is introduced as a means of providing some continuity to the various cycles. However, it 
need not be just theoretical. It can be made operational in ways that reflect a commitment 
to staff participation and empowerment in the management process. 

As a user of this monograph you are charged with the responsibility of sharing your 
experiences with a network of others so involved. 

jQue les vaya bien! 



George W. Mayeske 
Program Evaluation Specialist 
Extension Service 
U.S. Department of Agriculture 
Washington, D.C. 20250-0900 
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Foreword to the Second Edition 
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Chapter 1. Introduction to Life Cycle Program Management & Evaluation 

The topic of this monograph is Life Cycle Program Management & Evaluation. The 

Life Cycle term is borrowed from the extensive literature on project management (Cleland 
& King,1988; Kerzner,1989; 1984) where it is used to indicate the distinctly different 
stages that are experienced in the course of carrying out a project (e.g. the development 
of a product or the construction of a facility). This concept is proving useful as a 
framework for thinking about programs* as well, especially experiential educational 
programs (viz. nonformai educational programs that emphasize learning by doing). 

1.1 The Life Cycle for Educational Programs 

Figure 1.1 portrays the cycle that has been developed in working with experiential 
educational programs that are carried out by the Cooperative Extension System (an 
organization about which more will be said later). The milieu in which this cycle occurs 
as well as the nature of the different stages are: 

o Futuristic Perspectives lor the Organization 

Organizations usually sponsor a number of different programs. Consequently, the 
organization's view of where it is going in the future and how it will be successful form an 
important part of the climate in which programs are initiated and carried out. For this 
reason these concerns are portrayed as a background condition for all life cycle 
management. More will be said later about how futuristic perspectives can be obtained 
and how they might "optimize" the life cycle management process. 

o Problem Finding 

Problems may appear suddenly or emerge over a longer period of time. Efforts to 
ameliorate them may be imposed on an organization by forces external to it or may be 
initiated by the organization itself, perhaps as part of its routine functioning. Problem 
amelioration provides the impetus for programs. The effort devoted to problem 
amelioration by an organization will depend upon its mission and the importance of the 
problem relative to others the organization has to deal with. 

o Program Design 

The design of a program may be thought of as the development of a "blueprint" or "map" 
that lays out the structure of a proposed program describing what will be done, when, by 
whom and for whom, with a particular range of resources. It is in a sense, a theoretical 



*We have chosen to use the term "program" in lieu of "project" because many public 
sector agencies have programs which sponsor or fund projects. However, many of the 
concepts and principles are applicable for both programs and projects. Indeed, a project 
can be thought of as a particular type of program (see Chapter 14). For a related 
approach see Pancer & Westhues (1989). 
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£5ra > wmmitments are made, then the next stage is undertaken, 
o Program Development 

o Program Implementation 

implementation milestones. 

o Program Maintenance and Improvement 

manner. 

o Program Redirection 

to date. 
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1.2 Life Cycle Management as a Dynamic Process 

The notion of a program life cycle that encompasses different stages of development 
emphasizes the dynamic nature of the program management process. Programs are not 
-just static entities that once put in place-can be forgotten to let run indefinitely on their 
own. Rather, they have to be nurtured on a regular basis in order to be brought to a 
desired level of performance and stage of maturity. 

Before going on to define what is meant by a program and program planning let us 
examine a process by which this nurturing can take place. 

1.3 Life Cycle Management Through Team Guidance and Stakeholder Involvement 

In the life cycle process, programs that are identified are those that will involve the 
collective efforts of a number of persons. An explicit plan is developed in order to serve 
as a form of guidance for them in carrying out their specialized roles. For this effort a Life 
Cycle Guidance Team is formed. Its composition may vary depending upon the stage 
the program is in. In Figure 1 .1 , the role of the Life Cycle Guidance Team is portrayed 
as continuing throughout the cycle. An additional form of guidance is obtained from 
stakeholders. The definition of a stakeholder will vary depending on it's use. Generally, 
stakeholders are defined as anyone who can affect or be affected by what an 
organization does (Bryson,1988; Benveniste,1989). As applied to the Life Cycle Process, 
a stakeholder is defined as a person (or group) who has a special interest in or influence 
over a programmatic or topical area and who can provide useful information to the Life 
Cycle Guidance Team about the topic, program, stage that the program is in, or about 
the entire life cycle 'A a program. Their involvement in the life cycle process is a 
continuing one, as depicted in Figure 1 .1 . However, the nature and extent of their 
involvement will vary with the stage that the r ./am is in, as we shall see in later 
chapters. 

. Throughout all stages, the central task of the Life Cycle-Guidance Team is to sustain 
the plausibility of the program. Plausibility is defined by Smith (1989a) as H a judgment 
about the extent to which necessary and sufficient conditions exist for a program to 
succeed". 

1.4 What is a Program? 

A program is a theory which relates a set of organized activities and resources to 
intended results. Figure 1 .2 portrays the nature of this relationship. It is a system of 
beliefs or hypotheses derived from research, past experience and expert judgment. As 
the program matures through its stages this theory takes on an appearance of "reality". 
By this is meant that as one begins to see results emerge they are less inclined to regard 
the program as a set of hypotheses and are more inclined to think of it as being an 
"actual" set of relationships. Such observations notwithstanding, it seems advisable to 
always regard a program as b system of beliefs or hypotheses since uncertainties* always 
exist and a degree of uncertainty is more likely to keep our attention focused on it (things 
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can go wrong!). This definition is also a working definition which will be reformulated for 
greater specificity in subsequent chapters. 



Figure 1.2 Portrayal of Program as a Theory 
Linking Resources & Activities With Intended Results 




The skeptical reader might ask whether all this effort devoted to planning and 
management is worthwhile. Doesn't it after all, take valuable time and resources away 
from serving clientele? Also, doesn't this kind of a management process (viz. use of an 
explicit plan) stifle individual initiative and resourcefulness? 

The answer to the first question is that one must be aware of the nature of the 
circumstances in which they are operating. If the conditions are in such a state of flux that 
tomorrow and the day after may be very different from one another, then it makes little 
sense to plan as if they were going to be the same. However, this does not mean that 
one cannot plan for their being different (Sadowske.1991). In short, "failing to plan may 
be planning to fail". There are however, circumstances which demand immediate action 
and require little or no coordinated activities by different individuals. In such 
circumstances the time devoted to the development of a plan could divert energies from 
where they are critically needed. 



*We are reminded that most hypotheses are probabilistic rather than deterministic in 
nature. 
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For the second question one must recognize that no plan can be a substitute for 
individual initiative and resourcefulness. Every plan should be a guide to thought and 
action and not a substitute for thought and action. If a plan becomes an impediment then 
it defeats its purpose and should be discarded. 

1.5 What is the Cooperative Extension System? 

The motivation for this monograph as well as many of the examples grew out of work 
done in the Cooperative Extension System. Hence, it will be helpful in understanding what 
follows to examine the nature of this organization. 

The Cooperative Extension System is a partnership of Federal, State and county 
governments. It is administered by the U.S. Department of Agriculture and the land-grant 
universities. Its purpose is to provide educational programs oriented to the needs of local 
citizenry which are based upon the results of research. The topical areas dealt with are 
diverse ranging from: enhancing the viability of American Agriculture; wise management 
of our natural resources; improving the nutrition, diet and health of our people; helping 
families cope with changing economic and social circumstances; helping youth become 
productive and contributing members of society; and, helping to infuse a new vitality into 
the economic and social life of rural America. 

Most of the 16,000 professional staff of the Cooperative Extension System work at the 
county level throughout the States and territories. In a given year, these employees work 
with nearly three million volunteers and in so doing reach forty-eight million others-be 
they men, women, youth or ethnic minorities located in rural or urban settings. The extent 
of services may vary from a brief three minute phone call from a home owner concerning 
a horticultural or nutritional question -to working with a farmer for some years on a 
demonstration agricultural project -to working with the youth of a volunteer led 4-H club 
for several years. 

The funds to foster such a system come from each of the-three partners. Programs are 
usually planned at the local level for and with people of that locale with guidance from the 
other partners. Subject-matter specialists from the land-grant campus assist county staff 
in devising and delivering programs oriented to local needs based upon the most current 
knowledge available. When necessary these specialists or their colleagues will perform 
research needed to better serve these local needs. The States provide counties with a 
framework within which they can plan their programs and report on their 
accomplishments. The Federal partner performs a similar function for the States. County 
staff are accountable to both county and State administration while State staff are 
accountable to both State and Federal administration. 

The Cooperative Extension System is guided by a committee comprised of Administrators 
from the State and Federal level. It is called the Extension Committee on Organization 
and Policy (ECOP, for short) and is convened periodically in each year to deal with 
systemwide issues, problems and policies and to deliberate as to the roles the different 
partners should play in such matters. This committee with its various substantively 
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oriented subcommittees (e.g. legislation, budget, personnel, other topical areas) forms the 
governance structure for the Cooperative Extension System. Cooperative Extension as 
a nationwide system may be thought of as a widely dispersed, loosely coupled 
organization with multiple authority structures (Benveniste,1989: Brazzel and Sanderson, 
1990; Louis & Sieber, 1970; Weick,1976). The more than 75 year history of this system 
has been described recently (Rasmussen,1989; Mayberry,1989) and an examination 
made of its functions as a system (Mayeske, 1990; Bennett, 1992, 1990).lts' public 
familiarity has been studied (Warner & Christenson, 1984) and the history of its' most 
famous program - 4-H - has been described (Wessel & Wessel, 1982). 

Extension can also be characterized as a matrix organization (Kerzner,1989; Cleland & 
King,1988). By this is meant that while staff may have their disciplinary base in one area 
such as Agriculture or Home Economics, they may find themselves working in a number 
of areas outside of their disciplinary base. In so doing, they are accountable to different 
sets of clientele and perhaps supervisors, as well. For example, Agricultural Agents may 
find themselves working a portion of their time in Youth Development wherein they draw 
upon their disciplinary training and experience in their youth work (e.g. animal projects). 
In a similar vein, Home Economists may find themselves involved in nutrition education 
work with youth or in water quality problems with homeowners. Or, a staff person may 
have a joint appointment in Extension and Research wherein different functions are 
performed and are held up to different criteria of performance. 

In subsequent chapters we examine in more detail the stages of the life cycle process 
and many of the concepts touched on here. 
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Chapter 2. Futuristic Perspectives for the Organization 

Organizations will have perspectives about their future. These perspectives may develop 
through some formal, explicit means or informally, in a happenstance manner. If 
- developed in the latter manner, then there may be a wide diversity of views rather than 
a shared view arrived at through some consensus developing procedures. In addition, 
they may encompass some admixture of both near term and longer term views about the 
organization's future. However formed, members of the Life Cycle Guidance Team bring 
these views to their team setting and such views can have a profound effect on their 
enthusiasm for participation as well as the quality of what they produce. It seems evident 
that the team's work is best served if team members have a clear sense of how their 
organization will function in the future and of what roles they will play in that future. Such 
concerns become particularly salient when we focus on the final stage of the life cycle in 
which program redirection and endings are dealt with. Two complementary methods for 
developing a consensus on these perspectives are "strategic or longer term planning" 
and "near term work planning". 

o Strategic Planning 

Strategic planning is defined in somewhat different ways depending on the nature of the 
organization. For public sector and non-profit organizations a commonly used definition 
is "a disciplined effort to produce fundamental decisions and actions that shape and guide 
what an organization (or other entity) is, what it does and why it does it" (Sryson,1988)*. 
If an organization has engaged in strategic planning and if its employees are 
knowledgeable of such results (better yet if they participated in producing such results) 
then one can reasonably anticipate that the team members will bring to their work a better 
formed, articulated and" shared "sense of their organization" and how it may succeed 
in future years. Plans developed by such means usually deal with a longer term view - 
say 5 to 10 years into the future. They may of course, be renewed every few years. 

o Near Term Work Planning 

Yet another way in which futuristic perspectives are gained is through the development 
of specific near term (e.g. from several months to a few years) work plans. Such plans 
identify specific programs that will likely incorporate considerations from strategic 
planning, are subject to amendment periodically, and are the plans against which 
individual and group accomplishments are reported, also periodically. Employees at all 
levels of the organization are involved in its preparation wherein goals, objectives, 
resources to be used, numbers of clientele to be worked with, expected results and 
measures of these results are all specified. Employees involved in the preparation of such 
a plan will have a well developed, near term perspective about the nature of the 
organization and of their role in it. 



* For other definitions see: Armstrong (1985); Boyle(1981); Benveniste(1989); 
Coates(1986); Sadowske(1991); and, Simerly(1989). 
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Figure l.\ The Role of Futuristic Perspectives 
in the Life Cycle Process 
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0 Stakeholder Viewpoints 

A critically important source of information for the development of near term and longer 
term plans is stakeholder viewpoints. Just as programs, programmatic topics or 
thematic areas can have stakeholders, so too can an organization. At the organizational 
level the more general definition of a stakeholder is "anyone who can affect or be affected 
by the future of an organization" (Benveniste,1989; Bryson,1988). Stakeholders will have 
viewpoints about the future of the organization. Plans made by soliciting and 
incorporating these viewpoints into their development can certainly be made stronger than 
by leaving them out. Figure 2.1 portrays the influence of stakeholder viewpoints on the 
development of near and longer term perspectives and these in turn on the work of the 
Life Cycle Guidance Team. In the Cooperative Extension System (CES) examples of 
stakeholders are: university staff including the President, Deans, Extension administrators, 
other administrative staff, field staff, officials of other concerned or, cooperating 
organizations, organized interest groups, clientele, community leaders and elected 
officials, persons who assist in the provision of programs (e.g. volunteers), etc; By the 
nature of their involvement with the CES, stakeholders can provide useful viewpoints 
about the possible futures for an organization. 

This is by no means the only form of stakeholder involvement. As we shall see in 
subsequent chapters, they can be involved in a variety of different ways in the stages of 
the life cycle process. 
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Chapter 3. Problem Finding 



This chapter utilizes a general problem solving approach to focus on the general nature 
of a problem as well as how problems are found, described and possible solutions 
identified. Needs are seen as a special class of problems for which a solution exists. 
Different kinds of needs are identified as well as a taxonomy that allows for their 
classification. The roles of stakeholders and the Life-Cycie Guidance Team are discussed 
as are the pre-conditions necessary for the problem-solving process. Programs are seen 
as efforts to ameliorate or resolve problems. 

3.1 A Problem-Solving Process 

A problem is said to exist when a gap is perceived between an existing condition or 
state (the what is) and a desired condition or state (the what should be) (Van Gundy, 
1 988a). Figure 3.1 portrays the nature of a problem, if there is no gap then there is no 
problem. The terms perceived and desired receive special emphasis because they 
encompass the notions of value and pluralism. That is, what some persons perceive as 
a gap others do not because they hold a very different set of values about what is 
important. Desire also enters in because it involves not only the "what is" but the "what 
should be" as well. For example, the nature and magnitude of a gap will depend very 
much on what state or condition is desired, if there is a divergence of opinion about the 
desired condition(s) then no solution(s) may be possible. Further, conditions or states 
can change very rapidly so that they are best thought of as dynamic rather than static. 
Hence, Van Gundy (1988b) defines a problem as "a set of ongoing perceptions held 
about a constantly changing gap between a desired and existing state." By viewing 
problems as dynamic in nature one: avoids prematurely fixing on a symptom and a 
cause rather then considering multiple alternatives; is more open to consider new 
information since no boundaries have been established; and, is more likely to view the 
problem in a way in which adaptations to environmentai changes can be more readily 
made (Van Gundy, 1988b). The major disadvantage in using a dynamic definition is that 
it is more difficult to reach closure on a problem and develop a solution to resolve it. 
However, difficulty in reaching closure forces us to engage in a problem finding 
approach. The problem finding approach entails a series of redefinitions or alternative 
definitions of the problem until one redefinition of the problem is seen as a solution. This 
is called a problution "to symbolize the close relationship between problems and 
solutions" (Van Gundy, 1988b). 

Problems can be classified according to their degree o1 structure (Van Gundy, 1 988a) as 
follows: 

Well-structured - has all the information available to close the 
gap. 

Semi-structured - has enough information available to at least 
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partially define the nature of the gap but there is a lack of certainty about the 
actual or desired state or about how to close it. 

♦ Ill-structured - lacks information on how to close the gap. 

For the well-structured problem, routine solutions can be obtained readily. For the semi- 
structured problem, some combination of routine and creative solutions are required 
whereas, for the ill-structured problem, solutions have to be created as part of the 
problem-solving process.* 

A general problem-solving model involving the three phases of Intelligence, Design and 
Choice is given in Figure 3.2 (as adapted from Van Gundy, 1988a, p. 7). Examination 
of this model shows that many of the steps in these phases are the same as those 
involved in deciding whether'or not some form of an educational program, either alone 
or in combination with other efforts, can help to solve or ameliorate a problem. 



* Techniques for reaching such solutions are given by Van Gundy (1988a). 
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3.2 Sources and Techniques for Problem Searching 



Problems may emerge through a deliberate search or they may be thrust upon us by 
circumstances. Usually we try to identify problems in a systematic way so that we might 
better understand their origins. To do this we seek information from a number of sources 
using a variety of techniques or methods. Some techniques commonly used are: 

Scanning of key sources such as media coverage, 
documents, social indicators or through the observations of 
key "scanners" (persons who are strategically situated and 
committed to perform some form of observation) (Coates, 
1986). 

Surveys including mail questionnaires and interviews (phone 
and face-to-face) (Dillman, 1978). 

Structured Group Processes in which groups are especially 
formed, usually so that the group setting can foster the views 
expressed. Examples of some of these are: ' 

Hearings or forums in which participation by 
concerned parties can be voluntary or solicited 
and can utilize "town-hall" sessions or 
interactive TV link-ups (McKillip, 1987; United 
Way, 1982). 



Contrived groups in which a set of fixed, 
interactive procedures are followed to elicit 
views not usually obtainable by other means. 
Examples of these are focus groups (Krueger, 
1988; Stewart, 1990); nominal and Delphi 
groups (Moore, 1987) and structured problem- 
solving groups (Van Gundy, 1988a). 

investigative reporting in which a skeptical observer goes 
out to "scout" around to see what they can "dig up" (Douglas, 
1976; N. Smith, 1992). 

Events, which can be of two types: 

Extant - those already in existence (e.g., fairs, 
church gatherings, etc.) which can be "piggy 
backed" for purposes of gathering information; 
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Contrived - those which are especially created 
so as to gather information (e.g. devoting a day 
or week in which problems will be identified, 
United Way, 1982). 

Some of the sources from which information can be obtained are: 

Analysis of extant information such as that available from 
Census data, health or literacy surveys, archives or record 
systems, special studies, budget information on allocations 
and/or expenditures, etc. 

Solicitation of information or advice from extant groups or 
organizations formed for other purposes but from whom 
particular viewpoints would be useful or as in Extension, from 
ongoing, institutionalized advisory committees formed 
particularly for purposes of problem identification (Cole & 
Cole, 1983). 

Solicitation of expert opinion from key informants such as 
subject matter experts, elected officials and community 
leaders, affected or concerned clientele or citizens, staff or 
clientele of related topical areas or organizations, or others 
who are so situated that they bring a particjlar viewpoint 
judged to be useful. 

It is obvious from the examples that our distinction between techniques and sources is 
not always a clear one. For example, an event can be a technique when it is especially 
created or a source when it already exists. Nevertheless, these examples do serve to 
illustrate some possibilities. 



3.3 A Framework for Problem Analysis 



In any general problem finding effort a number of problem conditions are likely to be 
- identified. If they can be separated from one another then one can focus on each in turn. 
Alternatively they may be so closely related that they have to be considered together. 
In any case some framework or series of questions to guide the analysis is needed. One 
such, adapted from the health services area is given briefly below (Kettner, et. al. 1990) 
followed by some examples taken from work in the Cooperative Extension System: 

1. What is the nature of the condition or situation? 

This refers to the initial state in our problem model in Figure 
3.1. It entails an analysis and synthesis of all of the 
information collected using some of the different techniques 
and sources cited earlier. This analysis and synthesis may 
involve many of the myriad quantitative and qualitative 
methods available plus visuals, graphics and simple verbal 
summaries (Patton, 1990; Rossi et.al, 1985). In this step, 
care must be taken to insure that all viewpoints have been 
included and that the problem has not been labeled 
prematurely. 

2. How are terms defined? 

Explicit definitions with agreed upon meanings must be used. 
Terms should not be pejorative in nature or carry surplus 
meanings that could misdirect one's thinking. 

3. What are the characteristics of those experiencing the 
condition? 

Usually this entails the description of those experiencing the 
condition in socio-demographic terms, such as attributes of 
the entity (person or thing, animate or inanimate) involved. 

4. What is the scale and distribution of the condition? 

This question deals with how many are affected by the 
condition and where they are located. 

5. What is the nature of the threat from the existence of the 
condition? Who or what is at risk because the condition 
exists? What are the implications if the condition 
persists? 
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How widely is the condition recognized? Is wide 
recognition needed in order to mobilize support for 
dealing with the condition? 

Who defines the condition as a problem? Who might be 
in favor or oppose doing anything about the condition? 
Who stands to lose or gain by action or inaction? 

What is (are) the origin(s) of the condition? 

Is there a single cause or are there multiple causes? Can 
causation be identified at all? Can the condition be modeled 
using some causal or sequential chain of events so that 
points of intervention and/or prevention can be identified? Is 
enough known or is new knowledge or research needed on 
some aspect of the condition? 

Are there special aspects of the condition that should be 
recognized? Are there ethical or cultural aspects of the 
condition that need to be recognized? Are some groups 
of much or little influence involved? Are there ethnic or 
gender concerns that need to be addressed? Is there a 
tendency to "blame" individuals for system shortcomings 
or vice-verse (Rogers,1983)? 

The reader may think of even more questions that can be 
added to this list. In Table 3.1 we have subjected each of the 
hypothetical conditions to this set of questions as a way of 
illustrating how they can be applied. 
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3.4 Needs as Problutions 

We have tried to distinguish between problem identification and problem solution because 
there is a very strong tendency to jump to solutions prematurely, before the problem is 
well understood (Van Gundy, 1988a; Kettner et.al., 1990). The concept of need, is often 
regarded as defining a problem whereas it really focuses on what can be done about a 
Droblem McKillip (1 987) defines a need as "the value judgment that some group has 
a problem that can be solved" (emphasis authof s). Hence, needs can more readily fall 
into the category that Van Gundy (1988a; 1988b) has called "problutions" if they are 
based on a thorough understanding of the problem. If they are not based on a thorough 
understanding of the problem then they might best be classified as solutions in search 
of problems--a condition that may prevail more than most would like to admit. 

In any event, values are seen as playing a key role in defining not only what a problem 
is but what can be done about it as well. Since the needs concept is involved in problem 
solution we may want to explore it in more detail. Theorists differ on the interpretation 
of need and several different kinds have been identified (Kettner et. al., 1990; McKillip, 
1 987) In the psychological realm need is sometimes seen as an organic imbalance or 
deficit which leads to behavior designed to reduce the deficit (Bandura, 1 986; Tyler 1 971 ; 
Maslow 1954). Thirst and hunger are obvious examples which meet this definition. 
However, for the human make-up perhaps the best known needs are from Maslow s 
hierarchy Maslow proposes a hierarchy of needs starting with physiological suiyiva 
needs (e.g., food, clothing, shelter). Only if these needs are met at some minimum evel 
can needs at the next level be satisfied (e.g., safety and security). And only if these latter 
have been met can one deal with the higher order needs for love and belonging, self- 
esteem, self-actualization (viz., realization of one's full potential), knowing and 
understanding, and aesthetics. 

There are two points especially worthy of note about this concept of a hierarchy of needs. 
First some needs are more basic than others. It's obvious that it is difficult to learn if you 
are hungry or cold. Second, in order to satisfy the higher order needs, the lower order 
needs must continue to be met. Some theorists call these maintenance needs (Scnven 
& Roth, 1978). If such met needs suddenly become unmet, then efforts directed 
towards the higher order needs on which they are based may fail. 

Four different types of need have been identified (Kettner, et.al., 1990). They are called 
normative, perceived, expressed and relative. Normative needs involve the use of a 
standard or norms against which the nature of a condition is assessed. Usually, these 
standards are established through laws, customs, general beliefs or scientific research 
(e g children need at least X calories/day to grow properly or, no one should go hungry 
for a'period of 24 hours). Perceived needs refer to what people think or feel their needs 
are These can be very different from needs as judged by some objective standard. 
Expressed needs refer to those that are met or unmet. For example, for a given service 
such as inoculations, one can determine how many received the service (met) and how 
many did not (unmet). The notion of overmet needs can also enter in here (United Way, 
1982) wherein persons receive more of a service than is warranted. Finally, relative 
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need refers to the level of services that one group or community receives vis-a-vis other 
groups or communities and reflects a concern for-,equity. For example, the level of 
services for an affluent community compared to one of low-income. Although these 
conceptions of need are not necessarily mutually exclusive (e.g., expressed needs can 
be compared to a norm, to perceived needs or relative to those of other communities) 
they do serve to illustrate a variety of ways of looking at needs. 

Connors (1992) identifies 5 factors which affect the delineation of need: (1) the definer 
of need (self vs. other); (2) the target of change (individual vs. social structure); (3) the 
source of need (viz. responsibility for occurrence) (external vs. internal); (4) 
remediability (fixed vs. changing condition); (5) satisfier responsibility ( personal vs. 
public). Such a taxonomy can be useful not only in conceptualizing needs but in 
determining how they might be assessed and redressed as well. 

We have not yet mentioned what roles the life cycle guidance team and stakeholders 

will play in the problem finding phase. Before we do so, it is appropriate to deal with the 
issue of who conducts the problem finding activities. 

3.5 Who Conducts Problem Finding Activities? 

Ideally problem finding activities would be conducted by an organization that had 
specialized expertise in such matters but that would not be involved in or in any way gain 
from a particular solution. Such activities would be conducted at a very general level and 
involve a broad range of citizenry residing in a particular locale. This general problem 
finding approach would sort out the problems identified and their possible solutions and 
identify the various actors, agencies, entities etc., that would work on solutions for which 
they were best qualified. Although this general approach is known to occur, especially 
in rural communities, it is more usual to find targeted problem finding efforts. It is quite 
common to find specialized agencies seeking problems related to their specialization. For 
example, education agencies are usually the ones to find educational problems and their 
solutions, health agencies, health problems and their -solutions, etc. Due to its broad 
community focus, Extension quite often finds problems whose solutions involve a host of 
other agencies even though there may be an educational effort involved as part of the 
solution. 

3.6 What Roles Do the Life Cycle Guidance Team and Stakeholders Play in the 
Problem Finding Process? 

Depending upon who conducts the problem finding activities the life cycle guidance 
team may not be formed until very near the end of this stage when the various agencies 
and actors are identified who will work on the solutions to different problems or to 
different parts of the same problem. In this latter case, the team will be composed of 
staff who have expertise related to the topic and who represent different levels and 
divisions of the organization. For example, Extension would draw upon field and State 
staff, as well as administrators and specialists (perhaps researchers too). If the life cycle 
team is formed earlier in this stage it is likely that they would serve initially as 
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4. Designing a Program for the Problem: An Overview 



Now that a problem has been identified we can begin to devise a program to deal with 
it. In this chapter we introduce a set of concepts and procedures that enable an 
organization to develop the design of a program. In the next four chapters we describe 
these concepts and procedures in greater detail. We then introduce an illustrative 
program design that is based on extensive work in the Cooperative Extension System. 
This illustrative design not only serves to demonstrate in detail the concepts and products 
involved but also serves as the basis for the discussion of later stages as well. 

4.1 What Is Program Design? 

This section describes a process that has been devised to determine how a program will 
be developed and implemented and with what results. We have come to call this process 
program design-by which we mean a theoretical framework for describing the 
effects and consequences of a program as they are related to its development and 
implementation. It is in a sense the plan of a program plan (Ruchelman, 1985) or, if you 
will, a "blueprint" of a plan. 

Program design was developed in the Cooperative Extension System in response to a 
recognized need to have a more disciplined way of developing programs, especially for 
topics that cut across disciplinary boundaries. It is an outgrowth of techniques that were 
originally developed or adapted by evaluators to the evaluation planning process in order 
to make evaluation results more relevant and useful. It is intended to increase the 
likelihood that programs will be successful. 

The process employs two main concepts: program modeling; and, stakeholder 
viewpoints. In the former, models are developed of key aspects of how the program 
plan will be carried out in a sequential manner, by what staff and with what 
consequences, in schematic form. In the latter, viewpoints are obtained from persons 
who have a special interest in or influence over the problem area being addressed in 
order to better inform the modeling process. These two concepts are implemented 
through the efforts of a team of six to fifteen persons who have expertise in developing 
and delivering to clientele programs related to the problem area of interest. This is called 
a staff-centered approach. It has proven especially useful in bringing together and 
developing a consensus among persons who are separated due to boundaries 
established by geography, organization, disciplines and, in some cases, even 
personalities. 

This staff-centered approach is activated through the efforts of a group facilitator(s) who 
directs and moderates the efforts of the group as they work their way through a sequence 
of disciplined steps that take place in a series of workshop sessions spaced over a period 
of days, weeks or even months. The products as they result from these sessions, are 
codified and put in more readable form by an organizational contact person (or their 
designate). 
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TABLE 4.1 PROTOTYPE AGENDA OF A WORKSHOP FOR THE 
DESIGN OF PROGRAM XYZ 

Purpose: (1 ) to develop a program design for use as a guide to program development, 
implementation and evaluation; (2) to determine key stakeholder interests in the way the 
program might be (or has been) designed, developed, implemented and evaluated; and 
(3) to ascertain the implications of (2) for program design, development, implementation 
and evaluation. 

FIRST SESSIO N (2 DAYS^ 
DAYS # ACTIVITY 

1/81. INTRODUCTION TO CONCEPTS AND PROCEDURES WITH AN EXAMPLE. 

1/8 2. BRIEF DISCUSSION OF STAKEHOLDERS AND IDENTIFICATION OF 

GENERAL CATEGORIES. 
1+ 3. FORMULATION OF PROGRAM LOGIC MODEL, FUNCTIONAL AND 

INDICATOR COMPONENTS. 
1/2 4. DEVELOPMENT/ADAPTATION OF SPECIFIC STAKEHOLDER QUESTIONS, 

IDENTIFICATION OF INDIVIDUAL STAKEHOLDERS TO BE INTERVIEWED 

AND DEVELOPMENT OF LETTERS AND INTERVIEW PROCEDURES. 
DEVELOPMENT OF PLAN FOR: CONDUCT, TRANSCRIPTION 

AND ANALYSIS OF INTERVIEWS; REVIEW AND ANALYSIS OF 

DOCUMENTS (AS RELEVANT). 
1/8 5. CONTINUATION OF MODELING (AS TIME PERMITS): IDENTIFY 

RESOURCES. BARRI ERS. BARRIER REDUCTIONS AND SPINOFFS. 

PLAN is conducted in 2 to 3 intervening months by organization responsible for 
Program XYZ 

3ECQMD session (2 davsi 

DAYS I ACTIVITY 

3/4-1 6. REVIEW AND SUMMARY OF INTERVIEW RESULTS. 
3/4-1 7. REVIEW OF MODELS AND COMPLETION OF MODELING. 
1/4 8. DEVELOP CONCLUSIONS, RECOMMENDATIONS AND NEXT STEPS; 
CONDUCT ADMINISTRATIVE BRIEFING(AS APPROPRIATE) 



Report is prepared by the organization responsible for Program XYZ 



WHO TYPICALLY ATTENDS THE WORKSHOPS: 

EG. 3-8 PROGRAM PROVIDERS WHO IMPACT DIRECTLY ON CLIENTELE 
2-6 PROGRAM ADMINISTRATORS &/OR SUBJECT-MATTER SPECIALISTS 
1 ORGANIZATIONAL CONTACT FOR WORKSHOP ARRANGEMENTS, 
INTERVIEWS AND REPORT PREPARATIONS 



Unlike the design team participants, the facilitator(s) does not need to be an expert in the 
subject-matter under consideration. Indeed, such expertise might conflict with the 
conduct of their duties. These efforts result in a program plan which is an agreed to 
product of the design team's efforts based on their collective knowledge and experience. 
It could not have been produced by any single member working alone. It enhances the 
likelihood of success of what will be done because those who are part of the team and/or 
their colleagues-those who must carry out the plan-have an explicit, agreed to guide to 
action. 

Table 4.1 displays a typical agenda for the workshop series. As is apparent from 
examination of this table, most of the work is done during the work group sessions. 
Typically, 2 sessions of 2 days duration are required. However, with some complex 
topics such as Water Quality, Youth Development or Leadership Development, a third or 
even fourth session might je required. The design team might also elect to hold a 
verification session in which program providers and subject matter specialists who were 
not part of the team are brought in to see how the modeling results conform to their 
thinking and experience. 

In this and subsequent chapters we expand upon the origin and use of the program 
design process and provide a number of illustrations based on experience in more than 
thirty different situations and topical areas. 

4.2 What Does the Program Design Process involve? 

In the early years of evaluation at the Federal level (late 1960's and early 1970's) many 
program evaluations were designed on the basis of high level managers conceptions of 
what the program was. Although expensive and time consuming some of these 
evaluations were inconclusive. Detailed examination of these programs showed that their 
evaluations were inconclusive because the programs were not being carried out in the 
way the managers thought or because there were not any programs. Consequently, the 
information collected was irrelevant. 

A technique called Evaluability Assessment (EA) (Wholey, 1979, 1987) was developed 
to determine if there was a "program." If so, then EA would help to determine what kinds 
of evaluation might be most useful. If not, then EA would help to determine how a 
"program" might be developed. 

EA procedures were developed with "top down" programs (viz., programs developed at 
higher levels to be carried out by lower levels). These procedures were adapted to the 
"grass-roots up" type of programming (viz., programs developed at the local level and 
aggregated upwards) conducted in the Cooperative Extension System, by Smith (1 989a). 
In using these adapted procedures it became increasingly clear that they were useful not 
only for planning evaluations but, with some modifications, for planning programs as well. 
These modified procedures, called program design are presented in this and the 
following chapters. 



For program design or evaluation, the preceding. procedures have been used successfully 
with the following program topics in the States/territories indicated: 



Before any design team activities are considered however, a major step is to obtain 
administrative commitment. This step may involve a good deal of time and effort, for 
some one individual or group must be persuaded that such a commitment of resources 
in terms of staff time, travel and interviews is worth the effort for the particular topic. This 
person may be the State Extension Director, the Administrative Council or a State Leader 
for a particular topic such as Youth at Risk, Food Safety, Waste Management, Water 
Quality etc. [Often the person doing the persuading has been a State Evaluator working 
in concert with their counterpart from the Federal office.] Others may have to be 
consulted before a decision is made. If a decision is made in the affirmative then an 
organizational contact person must be named, the design-team members selected and 
dates set for the first meeting.** 

The selection of design team members is an absolutely critical decision which will affect 
the success of the entire effort. The decision has two aspects: who--in terms of 
knowledge and experience; and, how many. For the latter there is no hard and fast 
number. Since the process entails a great deal of interaction among members-the larger 
the group the more discussion that is required-hence, the greater the amount 



* Topics dealt with initially by Smith (1989a) 

"Usually when a decision is made not to proceed the most frequent reason given is that 
there is no staff person who has the time and appropriate experience to serve as the 
organizational contact. A second reason is that there are often already excessive 
demands on the staffs time and a third, that the timing is not right for where they 
are with respect to the topic. 



Teleconferencing for Locally 

Elected Officials* (IL) 
Master Gardener* (CA) 
Aquaculture* (TX) 
4-H Youth Development* 
(MD, PA, MS, CT, LA) 
Home Based Business* (OK, AL) 
Family Well Being (DE) 
Economic Development (AR, KS, 

MO, IA, NE) 
Rural Revitaiization (DC: USDA) 
Water Quality* (MD, VA, CT, LA) 
Community Resource Development 

(American Samoa) 
Adolescent Pregnancy & Parenting 

(NCA&T) 



Child Care (ME) 

Leadership Development (ME, LA) 
Training Programs for the 

Elderly (ME) 
Middle Management 4-H (HI) 
Import Substitution (Guam, 

Northern Marianas) 
Growing Your Own 

Food (Micronesia) 
System for Planning 

& Reporting (KS, IL) 
Families With Young Children 

at Risk (1890 Consortium: MO, 

TN, MS, AR, AL, NC, VA) 
Area Agents/Specialists (NC) 
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of time required. To meet the needs of an experience and organizational mixture about 
6 to 12 persons are usually involved--a smaller number would run the risk of lack of 
credibility and a larger number could be unwieldy. It is an absolute requirement that 
some of the members, preferably a majority, are program providers who work with and 
- impact directly on clientele or potential clientele, in the topical area under consideration, 
broadly defined. For example, if the topic involved youth but had never been dealt with 
before, we would want staff who work directly with youth of that age to be involved. The 
remainder of the group* is comprised of subject-matter specialists for the topic of concern 
and administrative staff. The involvement of a high level administrator may demonstrate 
the importance that is attached to the effort and hence have a salutary effect on the 
motivation of the design team-provided of course that their presence does not inhibit the 
functioning of the group. The program providers on the group serve as "reality filters" to 
insure that what is proposed is practical or "do-able." The staff named to the group 
should be ones who will be able to participate in all of the meetings. Absences and ins- 
and-outs can be extremely disruptive and should be minimized (or eliminated in the case 
of ins-and-outs). Isolation of the group from their ordinary activities is very 
desirable. 

Finding a time when all of the design team members can get together may be difficult and 
may require holding the first meeting two to three months after the assignments are 
made. Similar problems may be encountered for subsequent meetings.Finally, a person 
needs to be selected to serve as the workshop facilitator. This person must be 
experienced with respect to the program design process and should preferably have 
some training and/or experience in program evaluation. The facilitator must be a third 
party to the topic of concern and preferably should be a third party to the organization 
itself. Experience has shown that the work group members are more inclined to attend 
to the tasks at hand if the facilitator is not "one of their own." Then too, by being a non- 
expert the facilitator can ask many "dumb" questions that can be revealing or even 
challenging without threatening the design team members. Two facilitators reduces 
fatigue and increases variety for the team. 

Once the design team has been convened and some introductory preliminaries and 
administrative endorsements dealt with, the activities in Table 4.1 can be initiated. A 
handout of materials is used by the facilitator to introduce the group to the process (see 
Appendix A). The cover sheet, like that in Table 4,1 , is discussed by the facilitator in 
sufficient detail to provide some clarification and incentive. Next, major concepts are 



*Experienced volunteers &/or potential clientele might also be included if such 
participation can be deemed meaningful & productive. For example, in Maine's 
Leadership Development effort they included an experienced volunteer who participated 
fully & made excellent contributions (Killam,1991). Similarly, for their project on 
Adolescent Pregnancy & Parenting, NCA&T included some teen parents in the initial 
session, with excellent participation, especially by the girls (Wade, 1993). If appropriate, 
members of other organizations or agencies might be included on the team, especially 
if it would be important to their agency's involvement later on. 
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introduced starting with a brief explanation of stakeholders and program modeling. Other 
key concepts are introduced and explained briefly through illustrations of actual and 
- generic models, stakeholder identification and generic questions, interview guidance, etc. 
A brief history of the development of the program design process is discussed as well as 
the benefits that derive from the process. An actual example report is then given, also 
as a handout (see Mayeske, 1 991 ), which serves to familiarize the group with one of the 
major products. 

Since these concepts are developed and expanded upon in subsequent chapters they will 
be dealt with in only a cursory manner here. With respect to program modeling the steps 
are as follows: a set of major or main events can be identified which comprise the 
program, its effects and consequences and which are sequentially and causally related 
to one another such that if one event fails to occur then all of those succeeding it in the 
causal chain also fail to occur (the program logic model); for each main event of the 
program, a set of activities with a corresponding set of resources, can be identified which 
must be accomplished in order for the main event to occur (the functional component); 
for each activity in the functional component one or more sources of evidence of the 
occurrence of that activity can be identified (the indicator component-used also for 
effects and their consequences); things happen that can perturb or disrupt the causal 
relationships (called barriers) but can perhaps be overcome by special efforts (called 
barrier reductions); things happen once the program effects have occurred which 
perturb or prevent the consequences from taking place and are difficult or impossible to 
overcome by special efforts (called intervening events); and, for the occurrence of each 
main event in the program logic model, unplanned effects may also occur which can be 
positive or negative, known or unknown (called spin-offs). In order to inform this 
modeling process as well as other aspects of the program design process, information 
is collected from a judgmental sample (Patten, 1990; Henry, 1990) of stakeholders 
concerning their views about the nature of the problem, issue or need and how it should 
be addressed, and by whom. 

Once the preceding concepts have been discussed the facilitator introduces the concept 
of stakeholders, gives a working definition: an individual or group who has a special 
interest in or influence over the topic or program-to-be ami who can provide 
information that will be useful for the design, development, implementation and 
evaluation of the program; and asks the group to identify some general categories of 
stakeholders. This brief exercise helps to "map the environment" or delineate spheres 
of influence/concern for the topical area or program-to-be. It usually produces too many 
categories, some of which may be of questionable relevance or utility. Rather than deal 
with that then, the facilitator usually waits until «a later stage when some of the modeling 
has been completed and the group has a better sense of what they are about. 

The modeling is then initiated by starting with the development of the matrix of program 
effects. We have dealt primarily with educational programs so we call it the Matrix of 
Educational Effects. The group identifies the target audience(s) for the program to be 



4-6 



40 



and then systematically completes the cells of the matrix using categories from the 
Bennett hierarchy (1979) of: Knowledge, Attitudes, Skills, Aspirations and fiehavior (or 
practice) changes. This matrix of effects (E) is then used as the basis for identifying their 
consequences (C) and antecedents (A). That is, what events logically follow as a result 
of E and what events must logically precede E in order to insure that E occurs, depicted 
as: 

A >E >C 

and read: if A then E, if E then C. Once all of these events have been identified then 
the activities/resources are identified for the A events and the indicators are identified for 
all of the events in the program logic model. 

The modeling process is interrupted in order to deal in more detail with stakeholders. 
Some generic questions are reviewed and either adapted or new ones are developed that 
are appropriate for the topic. These may be further refined or tailored to different 
categories of stakeholders once they have been identified. A list of specific stakeholders 
is then identified, a contact letter is developed as well as other interview procedures. 
Relevant documents such as task force reports, evaluation studies of related topics, 
program plans, etc., are identified at this point. If there is a need to review them, 
because no one on the work group is familiar with them, then some members of the 
group need to be given the assignment of orally reporting them at their next meeting so 
that the results can be used by the group in their deliberations. Upon completion of this 
step the team resumes the modeling process until it is time to recess. 

The recess period usually lasts 6 to 12 weeks during which time the organizational 
contact person sees to it that the interviews are conducted, and transcribed. The length 
of this period is usually determined by the need for time to complete the stakeholder 
interviews and the calendars' of the group members. On meeting again, the group 
divides up the interview results into groups of stakeholders with at least 2 members 
reviewing each of the sub-groups, discussing them with one another to reach agreement 
on their meaning and making some cryptic summary notes. These cryptic summaries are 
then reported to the full group and entered into a matrix format of question answers by 
stakeholder groups (on flip chart papers) by the facilitator(s). After reviewing and 
discussing these summaries, the group makes some general thematic observations about 
their results and implications. The group then reviews and completes the modeling 
started in the first session. Finally, the group makes some conclusions and 
recommendations for administration and a report of all the group's work is prepared, 
usually by the organizational contact. A briefing of the top level administrator(s) by the 
team may be included in these efforts. Among other uses, this report serves as the 
"blueprint" for program development and implementation. 
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4.3 What are the Benefits from the Program Design Process? 

The results of this process increase the likelihood that later efforts will be successful by: 

o involving staff in the process by giving them the time and the opportunity 
to meet together and reach a consensus on the "blueprint"; 

o providing an explicit causal framework for articulating the program's 
nature, effects and consequences; 

o identifying things that can go wrong and what might be done about them 
before they occur; 

o identifying unplanned results from carrying out the program in a particular 
way; 

o specifying sources of evidence that can be examined and/or obtained to 
judge adequacy of program implementation and degree of program impact; 

o involving persons of influence with respect to the topic and obtaining 
guidance from them before the program is developed; 

o clarifying to administration the nature of their commitment, especially 
with regard to resources; 

o enhancing the likelihood of resource commitment through an explicit, 
agreed upon plan that encompasses stakeholder viewpoints; 

o providing an effects oriented guide for program development; 

o providing a framework to communicate the program to others; 

o providing an agreed upon "blueprint" for future action; 

o increasing the acceptance of measured effects once the program is 
implemented, as resulting from the program. 

The process also has some direct benefits to the staff in terms of improving their program 
planning and evaluation skills and, to the organization in terms of increasing its visibility 
with stakeholders for the topic of concern. 

Figure 4.1 gives a graphic summary of some of these steps and benefits. 
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Figure 4.1 Graphic Summary of Steps in 

& Benefits from the Program Design Process 
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4.4 What Role Does the Life Cycle Guidance Team Play in the Program Design 
Process? 

The Life Cycle Guidance Team (LCGT) often has its origins in the membership of the 
program design team. It is not unusual for a subgroup of these members (say 2 to 4) to 
continue on into subsequent phases and form the nucleus of the LCGT. Hence, they 
provide not only linkage expertise with earlier phases, but actually take on some 
developmental and management functions for the program. The LCGT is augmented by 
other members who may serve for shorter periods of time amd who often bring some 
particular expertise or administrative viewpoint appropriate for the particular stage that the 
program is going through. 

Figure 4.2 attempts to portray the functioning of the design team wherein they bring 
together their own extensive experience, the viewpoints of stakeholders, their knowledge 
of their organization and its future, and related information ( research & evaluation results, 
other documents) in order to develop the program design. As a result of their efforts, 
recommendations are made to administration which can involve some executive level 
decisions concerning the next steps. A commitment of resources may be required, some 
new policies or collaborations may have to be initiated or the "go-ahead" may be needed 
for the next steps of development, implementation &/or evaluation. 
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Figure 4.2 Program Design Procedures for Life Cycle 
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5. The Program Logic Model and Its Components 



- This chapter describes the development of the program logic model and its components. 
The model is, as defined in the preceding chapter, a set of causally and sequentially 
related main events that define the program and its consequences and conform to an "if- 
then" relationship such that for any event to occur, all those preceding it must have 
occurred first. The first step deals with the development of the Matrix of Educational 
Effects. Once developed this matrix becomes the pivotal main event in the logic model 
and is the point from which consequent and antecedent events are identified. A generic 
schematic of these concepts and their interrelationships is given in Figure 5.1. Upon 
completion of the logic model, the functional and indicator components are identified. 

5.1 The Matrix of Educational Effects: A Starting Point 

In his hierarchy of a chain of events for extension programs, Bennett (1 979) identifies two 
kinds of "near end" program effects: (1 ) KASA (knowledge, attitudes, skills, aspirations) 
changes; and, (2) practice (or behavior) changes. Since Extension regards itself as an 
agency that offers educational programs it seems reasonable to start by identifying what 
educational (or KASA and behavior) changes will result from participation in the program- 
to-be. They are called KASAB's and are used as a starting point in developing the 
program logic model. They are by no means the only starting point. Smith (1989a) 
suggests starting with an enumeration of the full set of activities involved in the program; 
however, this has proven to be extremely time consuming and difficult for the group to 
do. 

The work group is first asked to identify the target audiences or anticipated participant 
groups for the program. This step may require some discussion or alternatively, may be 
very easy. Usually more than one target audience is identified and on occasion, an 
additional audience may be identified as an afterthought once the group has gotten into 
more details of the process. Once these audiences have been identified a matrix, like 
that in Figure 5.2, is put on the wall where all may see and the Facilitator asks the group 
to work their way through the matrix identifying the participant or clientele KASAB 
changes that will occur by virtue of partici^.ion, as exemplified in Figure 5.3. 

The KASAB's will require some discussion before the group begins to fill in the matrix and 
even during the process. The group needs to be reminded of the fact that the KASAB's 
are not usually something participants would acquire other than through the program. 
Some discussion points are: 

1. Knowledge: I, the participant, am now aware of conditions and have 
acquired factual information that I didn't have before the program. 
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Figure 5.2. Worksheet For Completing the 
Matrix Of Educational Effects 
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Figure 5.3 The Nature of Entries for 
Completing the Matrix of Educational 
Effects 



Target Audiences 



KNOWLEDGE 



I, THE PARTICIPANT, AM NOW AWARE OF CONDIT- 
IONS AND HAVE ACQUIRED FACTUAL- 
INFORMATION THAT I DIDN'T HAVE BEFORE 

THE PROGRAM 



ATTITUDES 



SKILLS 



X, THE PARTICIPANT, BELIEVE THAT THE 
THE CONDITIONS CAN BE CHANGED WHEREAS BEFORE THE 
PROGRAM I BELIEVED THE OPPOSITE OR HAD NO 
PARTICULAR BELIEF AT ALL WITH REGARD TO THE TOPIC 



I, THE PARTICIPANT, AM NOW ABLE TO DO 
CERTAIN KINDS OF ACTIVITIES AND/OR 
PERFORM CERTAIN KINDS OF FUNCTIONS 
THAT I WAS NOT ABLE TO DO BEFORE THE PROGRAM 



ASPIRATIONS 



I, THE PARTICIPANT, NOW WANT TO CHANGE 
CERTAIN CONDITIONS RELATED TO THE TOPIC 
WHEREAS BEFORE THE PROGRAM I HAD NO SUCH DESIRE 
OR EVEN HAD NEGATIVE FEELINGS ABOUT THE TOPIC 



behaviors/ 
"practices 



X, THE PARTICIPANT (OR FORMER PARTICIPANT) 
ACTUALLY DO CERTAIN ACTIVIT- 
IES OR PERFORM CERTAIN FUNCTIONS THAT 
I DIDN'T DO BEFORE THE PROGRAM OR PERFORM CERTAIN 

FUNCTIONS DIFFERENTLY DUE TO THE PROGRAM 
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2. Attitudes: I, the participant, believe that these conditions can be changed 
whereas before the program I believed the opposite or had no particular 
belief at all with regard to the topic(s). 

3. Skills: I, the participant, am now able to do certain kinds of activities 
and/or perform certain kinds of functions that I was not able to do before 
the program. 

4. Aspirations: I, the participant, now want to change certain conditions 
related to the topic whereas before the program I had no such desire or 
even had negative feelings about the topic. 

5. Behaviors/Practices: I, the participant (or former participant) actually do 
certain activities or perform certain functions that I didnt do before the 
program or perform certain functions differently as a result of the program. 

As the group begins to work its way through the matrix many entries will be generic in 
nature; however, the facilitator can probe to determine if there is some specificity to these 
even though they might be .too numerous or complex to write down. The KASAB 
categories will likely suggest changes Or effects that have never occurred to the work 
group before but which, once suggested, seem worthwhile. There is not any required 
order in which the matrix must be filled out. They can start with any row, column or cell 
of the matrix and jump around as they see fit. Usually, they wlU find it easier to start with 
a row and work across audiences. Sometimes they start with the B row first. The group 
should be reminded that each cell does not require an entry and, on occasion, an entry 
isnt even meaningful, as later examples will show. At times, knowledge and skills are 
inseparable and for such occurrences Ibid can be used to so indicate (e.g., Skills Ibid 
Knowledge entry or entries).* 

Before this point the group may have considerable discussion concerning whether or not 
the program is expected to result in B(ehavior) change. Almost invariably the group 
decides that the program does or should result in such change. This is especially so jf 
the group recognizes that "informed decisionmaking" is itself a form of behavior change.** 
For example, the aspiring entrepreneur who decides as the result of a 2-4 hour short 
introduction to Home Based Businesses that they will not go into business for 
themselves. 

The next question posed to the group is whether or not the B(ehavior) change could 
occur without the KASA changes occurring first. We make this distinction because they 
could 

* The matrix can also be used to prioritize the expected levels of effects for the different 
KASAB's and groups (see Chapter 14 on Special Topics). 

** See Carroll & Johnson (1990) for approaches to the study of decisionmaking. 
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occur simultaneously or, in a coercive program, even in the reverse order. Invariably the 
group decides that B change is dependent on KASA change occurring first. These then 
become our first entries in the Program Logic Model. 



Figure 5.4 Relationship of KASA Change to Behavior/ 
Practice Change for Each Target Audience's 
Educational Effects 



KASA 




BEHAVIOR/ 


CHANGES 




PRACTICE 






CHANGES 



Illustrative examples of Educational Effects Matrices are given in Tables 5.1 and 5.2. 

The first example is from work done in the Commonwealth of the Northern Marianas 
Islands (CNMI) located just north of Guam. The CNMI received Commonwealth status 
in the early 1980's and consequently their Extension Service is relatively new. The 
Extension Service of CNMI wanted to initiate a program that would help build up island 
agriculture thereby reducing their dependence on imports while at the same time 
improving dietary intake by encouraging consumers to change their food purchasing and 
food preparation practices to more nutritious foods (e.g., from eggplant to bell peppers). 
The work group of about 10 staff members (almost the entire land-grant staff) identified 
4 target audiences of growers, traders, consumers and special interest groups as 
indicated in Table 5.1. The entries in the matrix reflect at a fairly general level, the 
changes that they believe the program (called High Nutrient Density Products (HNDP)) 
should try to bring about. 

The second example (Table 5.2) is from work done with Iowa State University on its' 
Retail Trade Program (Hammond, D. et al, 1989). This program had considerable past 
experience to draw upon and consequently the 6 design team members could readily fit 
it into the logic model framework. The 2 groups of intended participants are the proprietor 
& staff and community leaders. The latter are considered important in fostering a climate 
in which small businesses might succeed. 
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Table 5,2. Matrix of Educational Effects for Iowa's 

Retail Trade Program 



Target Audiences 







COMMUNITY LEADERS 


KNOWLEDGE 


Grater Knowledge of business management 
Better understanding of business environment 


Better understanding of business environment 


ATTITUDES 


Greater awareness of public's perception 
of own business 

Greater sense of col 1 active nature of 
oo— unity 


Changed perception of business oo— unity 
Greater sense of collective nature of co— unity 


SKILLS 


Improved business and people Management 

skills 


Improved ski 1 1s in cooperative organizational 
efforts 


ASPIRATIONS 


Aspire to increase sales 
Desire to stay In business 


.Desire for aore vigorous business sector 
Desire for expanded tax base 


BEHAVIORS/ 
PRACTICES 


Change in actual business practices 

Change in dealing with customer &/or 
employee relations 


Increased support for business community 

Increased volunteer &/or civic organizational 
activities 
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The Educational Effects Matrix (E) usually forms the first main event in the program logic 
model. If one portrays a very simple "if then" sequence of antecedents (A) and 
consequences (C) of E by arrows, as follows: 



A > E 



> C 



then one can take as the next step either the identification of the consequences (C) of 
E or its antecedents (A). Since some of A includes the program and its development, it 
is usually easier to start with C although there may be a good bit of jumping back and 
forth between the two as they are actually identified. 

5.2 The Consequences of Educational Effects 

There can be any number of direct consequences of B(ehavior) or practice change. 
Usually these fall into two main categories: (1 ) consequences that accrue to the program 
participant or alumnus, and, (2) consequences that accrue to larger entities with which 
they are affiliated (e.g., family, firm, community, etc.) Using the same worksheet 
approach we try to identify (1) first and then (2). There may be more then one main 
event for each and, as some of the examples will show the chain of events may get fairly 
long-a result which the group may want to revisit and simplify at a later time. The logic 
model begins to take on the following form: 



wherein those C's closer to E are of type (1) and those further away are of type (2). 
An ending point may be something that resembles the concept called "Improved Quality 
of Life." Somewhere in this part of the exercise members of the group may express 
some uneasiness about these re mote events being associated with what they as program 
providers are trying to claim as their impact. They need to be assured that such concerns 
will be dealt with in a later section of the modeling exercise where they will deal with such 
concepts as barriers, barrier reductions and intervening events. 

Examples of both consequences and antecedents are given in the next section. 

5.3 The Antecedents of Educational Effects: Introduction to the Program 

One can readily identify at least three main events which would precede the Educational 
Effects (E) of a program. They are: (1) the identification of a need, issue or problem 
which may have resulted from a needs assessment or issue identification process; (2) the 
development of a program to address them; and, (3) the implementation of the program 
(Boone, 1985). 



E 



> C 



>C 
2 



> C 
3 
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The workgroup usually starts with step (1) since they feel such a step should be taken 
even though that may not be the way they initiated other programs. This first main event 
is usually identified fairly readily.* However, in moving to the next main event the group 
will usually feel the need to include some events related to Extension identifying its role, 
reviewing available resources, etc. Such events can be either sequential or concomitant 
(viz., nearly simultaneous events) and should of course, conform to the "if-then" 
sequencing. A "rule of thumb" is to try to keep the models as simple as possible and 
conforming to the linear format since the Expanded Logic Model promises to be even 
more complex anyhow. 

There is much room for stylistic preference in how one proceeds to the next set of events. 
There are no apparent rules to follow as of this writing. The group may want a series of 
event boxes some of which run in parallel in order to reflect the complexities involved. 
Usually the group will settle for a much simpler model than the Facilitator is capable of 
drawing. 

Finally, prior to the "Provide Program" event the group may want to highlight intermediate 
events to which they attach some importance. Examples of these may be Recruiting and 
Training Staff and/or Volunteers, Implementing a Promotional Plan, Initiating a Network, 
Establishing a Research Linkage, etc. See Figure 5.5 for a generic example. 

A procedural note is with mentioning here. With most groups the program logic model 
is completed before work is started on activities and indicators. With other groups they 
are developed as each main event is identified. However, if it is the latter type of group 
they may get on a "roll" wherein events in the logic model seem to start falling in place 
very rapidly. In such a case it is often worthwhile to postpone the identification of 
activities/indicators until the events get "roughed in" and then cycle back to do them. 

On the following pages examples are given of completed program logic models. 



* Some groups may want to precede this event with statements, boxes, etc. containing 
the organization's mission, vision, functions, goals, objectives, etc. 
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Figure 5.6 shows the program logic model for the HNDP program in the Northern 
Marianas. This is a program that is very simple in its antecedents and very complex and 
rich in its consequences. It serves to illustrate a very important concept-that of mutual 
clientele effects. This is the case in which the educational effects achieved with one set 
of clientele have effects on other clientele of the program as well. At times this is 
intentional, as in the case of special interest groups such as elected officials (SIG's) 
adopting a position or viewpoint that is supportive of changes by other groups in the 
community (if not intentional, it is called a spin-off). These changes can occur 
concomitantly, reciprocally or sequentially. They can be represented by a single 
arrow between 2 events, concomitant if they occur nearly simultaneously and 
sequentially if they are separated in time. A double arrow (or arrows) is used to show 
reciprocal effects for 2 events that are (or are nearly) concomitant. 

The program logic model for Iowa's Retail Trade Program is given in Figure 5.7. In this 
program as well, there are some mutual clientele effects wherein small businesses are 
worked with to improve their practices while community leaders are worked with to 
improve the community conditions in which the businesses must operate. 

5.4 The Program Logic Model Functional and Indicator Components 

Once the main events of the logic model have been identified the group can proceed with 
the identification of each event's supporting activities and their indicators of 
accomplishment. The supporting activities are part of the functional component. The 
other part is the resources. The two are kept separate because the former serve as a 
basis for specifying or estimating the needed resources. The indicators are specified only 
for the supporting activities since they serve as a source of evidence that the activity has 
occurred or is occurring. It is also important to recall that indicators occur throughout 
the model. In the antecedent events (A) they relate to the occurrence of activities but in 
the effects (E) and consequences (C) they relate to the occurrence of E and C. A generic 
example of a program logic model with functional and indicator components is given in 
Figure 5.8. 

Often the group specifies the indicators when they complete each main event's activities 
or consequences. The worksheet formats used for each main event are given in Figures 
5.9 and 5.10. The facilitator merely puts a worksheet on the wall with the appropriate 
labels so that the group can start identifying activities and indicators. Moving back and 
forth from one main event to earlier ones to specify additional activities may occur fairly 
frequently. 

Resources are included as part of the functional component. However, their estimation 
by the group may be done anytime after the activities have been completed. Some 
groups may want to wait until they complete all other aspects of the modeling before they 
estimate the resources required. There may be some wisdom to this since other aspects 
of the modeling, especially barriers, barrier reductions and spin-offs, may suggest still 
other activities that could or should be done. 
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Figure 5.9 Worksheet for Determining 
Main Event Activities & Indicators 



MAIN 

EVENT 
#k 



Act I v I t 1 «t« 



Ind i cttors 



^Changes to Effects or Consequences for later Main Events 

Figure 5.10 Worksheet for Determining 
Resources to Support Activities 



MAIN 
EVENT 
#k 



Raaourca 
C a -t a g o r I a 



E a -t I ma-bad 
Full T I m§ 
Equivalent 



Ettlmattd $ 
Equ I va I an t. a 



5- 



The ease or difficulty with which resources can be estimated by the group will depend 
upon their prior experience in carrying out similar types of programs.* A typical 
worksheet for a set of activities might use the following categories in the worksheet format 
in Figure 5.10, and appear as follows: 

PROGRAM XY7 RESOURCES 
SAMPI F CATEGORIES 

STAFF CATEGORIES FTE's (Estimated^ $(EStimated) 

Professional Staff Time 
County 
State 
Specialist 

Para-professional 



Volunteer 
Secretarial Support 
TRAVEL 

FOtllPMENT/MATERlALS 

Acquire/Purchase 
Develop/Print 

OVERHEAD 

Once the appropriate resource categories have been identified, estimating the costs may 
not be difficult even though much discussion may be involved. Start-up, developmental 
and operational costs must all be estimated and usually there is a place for each in the 
logic model." 



* An exception is for modeling multiple programs simultaneously (see Chapter 14). 
** At times the group may want to postpone until another time the estimation of resources 

since they may not know what, if any, will be available (e.g., the number of locales in 

which the program will be carried out). 
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Illustrative examples of functional and indicator components are given in Tables 5.3 and 
5.4. 

The indicators for the educational effects (E) of the program and for its consequences (C) 
tend to differ from those for the antecedents (A). This is so because the indicators for 
A tend to be administrative documents while those for E and C may require some 
information collection that involves more than just the regular programmatic efforts. A? 
a consequence they may contain both methods (e.g., survey) and sources (e.g., agents, 
volunteers) rather than just sources. Table 5.5 gives the educational effects indicators 
for Iowa's Retail Trade program whiie Table 5.6 gives the list of consequences and 
indicators for Event 1 1 in the HNDP program logic model, Economic & Social Benefits. 
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Table 5.5 Examples of Educational Effects Indicators for 

the Program on Retail Trade 

Target Auulences 





PROPRIETORS & THEIR EMPLOYEES 


COMMUNITY LEADERS 


KNOWLEDGE 


Survey 

Agent Observation 


Survey 

Agent Observation 


ATTITUDES 


Survey 

Agent Observation 
Unsolicited Comments 


Survey 

Agent Observation 
Unsolicited Comnents 


SKILLS 


Survey 

Agent Observation 


Survey 

Agent Observation 


ASPIRATIONS 


Survey 

Agent Observation 
Unsol i cited Ccntnents 


Survey 

Agent Observation 
Unsolicited Comments 


BEHAVIORS/ 
PRACTICES 


• Survey 
Agent Observation 


Survey 

Agent Observation 



Table 5,6 Examples of Indicators for the Consequences 

of the HNDP Program 



Consequences 



ECONOMIC 
& 

SOCIAL 
BENEFITS 
11 



Longer I i f e span 

Reduced incidence of: hypertension, diabetes, 
heart disease, cancer, gout 

Improved dietary practices 

More viable agriculturally based enterprises 

Improved grower income 

Increased awareness & understanding of agriculture 
in Island economy 

Better Institutionalized support for Land Grant 
programs 



In d i cators 

Vital statistics 
Vital statistics 

Survey reports & records 
Cormerce & Labor statistics 
Survey, sales reports 
Executive orders, bills, laws 

Laws & appropriations 



74 



Table 5.7 Estimated Professional Staff Resou 
for the Program on Retail Trade 



Estimate of Staff Resources Expended: 


Estimate of Staff Resource Usage 


1 FTE state staff 


By Program Logic Model Events: 


1 FTE area staff 






1 FTE county staff 


.25 FTE 


Identification of needs 




.15 FTE 


Define roles 


3 Total Expended 


.10 FTE 


Identification of resources 




1.25 FTE 


Develop program 




1.25 FTE 


Implement program 




3.00 FTE 


Total expended 



Resources are the very last Item addressed, usually after all other asDects of th« 
modehng have been completed. However, they are introduced here as part o^l al£ 
.dent.ftcat.on process. The resource estimates given in Table 5.7 sfrve to tor^S 
way staff resources may be involved in the different main everts .VWh m^o\ner 



5.5 The Expanded Program Logic Model 

Completion of the logic model and its components is an important accomplishment for the 
group A th.s point they have a good idea of what they are abouTs^tha Mh ^exoans Sn 
^iSSST t0 inC ' Ude barri9rS ' barri9r ^udioJandinTe^^ 



to Ser in the to^SSS 9 VZ™ 9 W E 1 the W3y " in movjn 9 from one main ™* 
to anotner in the logic model. There may be th ngs that staff can do to surmount thesp 

bamers-these things they can do are called barrier reductionTlt's kind Z a vidlo 

*?J ° nS ^ thG barrier then one 9° es t0 barrio reduction ^an^ves through 
barrier reduction to the next main event. Schematically it is portrayedin R^re 5 11 The 
top arrows depict straight forward movement from event k to k + 1 tTi^ arraw 

tSSH if?o 9 K he bamer(S) ' 9 ° ing t0 barrier reduction ( s ) and then on to ev^" 
course, if the bamer is not surmountable then one is "out of business." " 

^I^^R 1 ^ 1° nU t mber th ° main events in the fc* model and then use 
me worasneet with the main events as a heading, as illustrated in Fioure 5 11 Thk 

^oTd^thJ'K 9 th3t f ° r 2" devel °P™ nt * the Functional mSi^S^^ 
To identify the bamers and their reductions the workgroup will want to be ableio refer to 
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Figure 5.11 Worksheet for Determining 
Barriers & Barrier Reductions 




I BARRIER | 
0| BARRIER | 

L 




(BARRIER I 
REDUCTION I 



r r I 



r m 



& m r r I « r 
Rtduct t o 



Figure 5.12 Worksheet for Determining 

Intervening Events 



EDUCAT- 




IONAL 




EFFECTS 






Intervenl n 9 Events 



the activities listed in the two events under consideration-especially the event they are 
trying to move from. The group works their way through the model considering each 
pair-wise combination of sequential events in turn until the Educational Effects (KASA and 
B) have been completed. In doing these it is not unusual for the group to discover that 
-there are some additional activities that they need to build into the Functional Component 
(thereby also amending the Indicator Component). 

When the Behavior/Practice Change event has been completed the nomenclature 
changes because this is usually regarded as an ending point for what the program staff 
can consider themselves responsible for or can claim responsibility for. It is not that the 
program does not want to affect later events but rather that other influences may come 
in to play that perturb their occurrence (or decrease their likelihood of occurrence), and 
the program staff has little or no influence over them. These are call intervening events 
and they are depicted schematically in Figure 5.12. The same worksheet format is used. 
However, it is often the case that indicators are not identified and sometimes there is only 
one intervening event. 

A generic example of an expanded logic model is given in Figure 5.13 while actual 
examples of barriers and barrier reductions are given in Tables 5.8 and 5.9. Examples 
of intervening events are given in Table 5.10. 
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Table 5.8 Examples of Barriers 8c Barrier Reductions 

for the HNDP Program 



DEVELOP 
EDUCAT'NL 
PROGRAM3 


| BARRIER | 
H BARRIER 1 


IMPLEMENT 
EDUCAT'NL 
PROGRAM^ 




1 f 





[REDUCTIONS 



Barr I o r s 

Lack of adequate staff time 

Conflict with other ongoing staff activities 
Lack of faci I ities/materials/resource people 



Barr I or Reduct 1 on* 



Set calendar far enouah In advance 
so staff can commit their time 



Communicate calendar to other 
agencies & to administration 



Seek adequate funding, promote 
oeve ' 



staff development 



Table 5.9 Examples of Barriers & Barrier Reductions 

for the Retail Trade Program 



KASA 
CHANGES 

6+7 


| BARRIER | ' 
M BARRIER | 


BEH/PRACT 
CHANGES 

6+7 









[REDUCTIONS 



Barr iers 

Lack of financial resources 
Lack of motivation 

Fear of risk-taking 

Perceived risks exceed benefits 

Insufficient priority 

Lack of desire to change 

Negative peer pressure 

It won't work here— we are different 

Tradition 



Ba rr i er 



Reduct I ons 



Educational or financial alternatives 

Use case studies, emphasize small successes, 
ascertain personal benefits 

Case studies help evaluate risk/benefit 
real istical ly 

Case studies help evaluate risk/benefit 
real istical ly 

Use peer pressure (band-wagon effect) 

Case studies reinforce benefits of change 

Obtain endorsements, use case studies, 
mobilize positive forces 

Case studies, peer carmunity visitations, 
estab 1 1 sh networks, involve negative 
sayers In decision, change camwnity perception 

Change camiunity perception 
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Table 5.10 Examples of Intervening Events 

for the HNDP Program 




r i 

M INTERVENING |i 
| EVENTS I 
L J 




Inter vening Events 

Natural disasters (e,g typhoons, flooding, 
drought, epidemics, climatic changes) 

Widespread violence 
Economic recession(s) 

Artificial disasters ( e,g, nuclear fallout) 

Economic boom leads to negative disruptions 
( e,g, farmers leave farming) 
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5.6 Identifying Spinoffs 

The final modeling exercise is one of identifying spinoffs. These are things that "just 
happen" because the program is structured and implemented in a particular way. These 
"happenings" may be good or bad, beautiful or ugly, known or unanticipated--they are 
unplanned occurrences that the group can identify.* For example, we know that in 
carrying out the 4-H Program, Agents and Volunteers serve as role models for youth; that 
in carrying out a program with volunteers they experience "psychic income" as well as 
time management problems; and, that para-professionals may go on to more education, 
better paying jobs or leadership positions in the community by virtue of their involvement 
in a program. We portray these schematically as in Figure 5.14. 

The worksheet format for these involves doing them all together in one matrix format as 
follows with a check mark or asterisk used to indicate which spin-offs pertain to which 
events as shown in Figure 5.15. Indicators are not always identified for spin-offs. 
Sometimes they are self-evident; usually time begins to get short and the group decides 
to move on to other tasks. Spinoffs usually reflect "growth"experiences that accrue to the 
organization, staff or participants by virtue of their association with the program.** The 
work-group usually finds them enjoyable and easy to do.[ A comprehensive list of spinoffs 
is given in Chapter 9.] 

5.7 Modeling Multiple Clientele Effects 

For the Program Logic Models used as examples in this chapter (Figures 5.6 & 5.7) we 
have noted the occurrence of different categories of clientele having an influence on other 
categories of clientele where this is a planned occurrence. As illustrated in Figure 5.16 
these influences can be concomitant - occurring at about the same time; sequential - 
one must occur before the other; or, reciprocal - the categories have effects on each 
other at about the same time. Figure 5.17 gives an example of a Program Logic Model 
which was developed for the Louisiana 4-H Youth Development Program (Richard.R. & 
E. Johnson, 1992) in which these multiple clientele effects are richly illustrated. 

5.8 Summary of the Steps Involved in the Modeling Process 

Figure 5.18 attempts to summarize the sequence of steps involved in the modeling 
process. The steps involving the specification of the educational effects indicators is listed 
separately because they are usually specified at the beginning of the second 2 day 
session, thereby serving as a review of what was done earlier. The final step involves 
estimating resources. It is connected by a dashed line to indicate that such estimation 
may be delayed until some administrative decisions have been made about resource 
allocations o r policy matters. 

* See Sieber (1 981 ) for a very thoughtful discussion of the unanticipated consequences 
of purposive action. 

** In a follow-up of participants subsequent to their work group experiences, it was found 
that they had developed a better understanding of the evaluation process & its use in 
planning & conducting educational programs (Johnson & Richard ,1993) 
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Figure 5.16 Causal Relationships Among 

Target Audiences 



CONCOM I TANT 







TARGET 
AUDIENCE 
A 


PROVIDE 
EDUCAT- 
IONAL 
EXPER- 
IENCES 




I 






TARGET 
AUDIENCE 
B 



SEQUENT I AL 



PROVIDE 
EDUCAT- 
IONAL 
EXPER- 
IENCES 



TARGET 
AUDIENCE 




TARGET 
AUDIENCE 
B 



RECIPROCAL 







TARGET 1 
AUDIENCE 1 


PROVIDE 
EDUCAT- 




I 


IONAL 


t 


EXPER- 




IENCES 










TARGET 1 




87 


AUDIENCE 1 
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Figure 5.18 



Summary of the Sequence of Steps 
the Program Modeling Process 



Involved in 



COMPLETE THE 
MATRIX OF 
EDUCATIONAL 
EFFECTS 



2 "DETB»jgCTHE 
OPTIONAL) 



DETERMINE THE 
ANTECEDENTS OF 1. 
THEIR ACTIVITIES 
& INDICATORS 



4. 



SPECIFY THE 
EDUCATIONAL 
EFFE CTS,. 
INDICATORS 



5. 



SPECIFY 
BARRIERS 4 

BARRIER 
REDUCTIONS 



IDENTIFY 
INTERVENING 
EVENTS 



IDENTIFY 
SPIN-OFFS 



B. 



ERIC 



ESTIMATE 
RESOURCES 
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5.9 Judging Plausibility 



- All program design and development efforts are intended to make programs more 
impactful once they have been delivered (viz., increase the likelihood that they achieve 
their intended goals and objectives). Short of demonstrated impacts (e.g., 
practice/behavior change alone or in combination with its consequences) as determined 
by evaluation results, one must rely on the concept of plausibility. Plausibility is a 
judgment about the likelihood that a program, either proposed or in different stages of 
its evolution (development, implementation, maintenance), will achieve its intended goals 
and objectives. All of the efforts devoted to modeling and stakeholders are intended to 
enhance the plausibility of a program. This plausibility judgment can be made by one or 
more experts or by stakeholders to a program. It is a judgment made many times by 
busy administrators using implicit criteria. In program design and development we 
attempt to explicitly identify a set of criteria to be used in making such judgments. 

Smith (1989a, p. 6) defines plausibility as "a judgment about the extent to which 
necessary and sufficient conditions exist for a program to succeed, i.e., are activities of 
the right type and amount to bring about the desired change?" and goes on to specify just 
what these conditions should be (pp. 115-123). In the following discussion we have 
adapted and expanded upon these conditions so as to facilitate their use for programs 
in their various stages of evolution. 

5.9.1 Plausibility Criteria 

Smith (1 989a, p. 1 1 5) asserts that necessary and sufficient conditions exist for a program 
to succeed if: "(1 ) it intends to bring about some change; (2) its intentions are clear; (3) 
its planned activities are reasonable i.e., they are of the right nature to influence the 
expected outcome; (4) its activities are sufficient in quantity and quality to exert that 
influence; (5) its resources are present in sufficient amount and type for the activities to 
be implemented as planned." To identity these conditions she poses a series of 
questions which in our adaptation are as follows: 

. o Do the main events have a clear and understandable purpose? 
oo Are they sequentially and causally related (does the 

IF— >THEN framework apply)? 
oo Are there means to judge their occurrence (see later 

question on activities)? 

o Are the educational effects clear and understandable? 

oo Are they sufficiently specific or do they lend themselves 

to further specification? 
oo Are there: (1) indicators; (2) criteria; and, (3) sources 

of evidence to judge their occurrence? If not, are there 

indications as to how they can be obtained? 
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Do the educational effects imply the consequences? 

oo Are there: (1) indicators; (2) criteria; and, (3) sources 

of evidence to judge their occurrence? If not, are there 

indications as to how they can be obtained? 

Are the supporting activities for the antecedent main events 
clear and understandable? 

oo Is each activity critical tc the accomplishment of the 
main event (viz., does lack of occurrence lead to 
non-accomplishment)? 
oo Are there: (1) indicators; (2) criteria; and, (3) sources 

of evidence to judge their accomplishment? 
oo Does the accomplishment of all lead to the accomplishment 
of the main event? 

Have resources been clearly specified as to the type and 
amount needed? 

oo Are the resources adequate to insure the accomplishment 
of the activities? If not, can they be obtained? 

Have barriers been identified? If so, are there ways of 
coping with them if they should occur? 

Have unplanned consequences of the program been identified? 
If so, are some of them so fraught with risks as to make 
program implementation unlikely or undesirable? 
oo Are some of them likely to work counter to the desired 

effects or consequences? 
oo Are some of them likely to exacerbate the problem that the 

program was intended to ameliorate*? 

Have intervening events been identified? If so, is the 
likelihood of occurrence of some of them so gwat as to make 
program implementation unlikely or undesirable? 

Is the program so structured that some members of the target group are more 
likely than others to participate in the program? If so, why? 



Sieber (1981) identifies seven mechanisms by which intentions are converted into 
opposite results. 
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In addition, for programs that are near the point of implementation, the following can be 
asked: 

o Is there evidence that clients will have the necessary 
prior knowledge and skill to benefit from the program? 

o Is there a high degree of correspondence based on logic, prior experience 
and/or research results, between the program content and the problem, issue 
or need being addressed? 

o Is the content of the program of sufficient breadth and 
depth to enable the client to: 
oo learn what is proposed? 

oo have enough contact with the program to acquire a 

sufficient level of skill to do what is proposed? 
oo know what else they must have (equipment, services, 

resources) or do to make the planned changes? 
oo believe that they can make the change and have the 

desire to do so? 



o Have provisions been made for: promotional efforts; the 
involvement of other agencies; and, the involvement of key 
influentials for program legitimation? 
oo Are these efforts appropriately tailored to the audiences 
involved? 



o Have provisions been made for staff training? 

o Have provisions been made for monitoring program 

accomplishments and their usje for program management? 



Undoubtedly experienced program 
Plausibility relates directly to how wel 



planners can think of even more questions, 
these questions can be answered. Since there are 
degrees of how well the questions dan be answered, plausibility is really a probabilistic 
concept-a program has a degree of plausibility rather then being or not being 
plausible. 1 



5.9.2 Conditions That Enhance Plausibility 



There are other conditions that also foster plausibility. They are robustness, 
consensus, and commitment-each is deserving of some discussion. 

In the course of interviewing stakeholders and involving them in the program design 
process a degree of support is generated for the program. In addition, those who have 
been critical of the agency's past efforts are allowed to air these criticisms and suggest 
constructive alternatives. Hence, though not intentional, stakeholder involvement can 
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lead to a kind of robustness in that the program may be better able to survive the throes 
of competition with others be it for resources, autonomy or authority. 

Another condition, which if present can greatly enhance plausibility is consensus-the 
extent to which there is agreement among those who are instrumental in developing and 
delivering the program as to what they are about and what their respective roles are in 
this effort. In the absence of such agreement it would seem problematic to expect that 
the results of their efforts would add up in any meaningful way and this is especially so 
if their roles are interdependent. The design team's involvement in the modeling process 
is an important means of developing consensus as are the verification exercises a way 
of developing consensus with those not part of the original team. Hence, staff consensus 
can be yet another important aspect of plausibility. 

Staff involvement also develops a degree of commitment to the program which can be 
a powerful motivator in all aspects of program design, development and delivery. 

5.9.3 Evaluability and Plausibility 

This is not a monograph on the do's and dont's of program evaluation, even though there 
are a few chapters devoted to the topic. However, it is of value while discussing 
plausibility to examine the notion of the evaluability of a program, especially since many 
of the techniques used in program design and development have their origin in that 
discipline. Put very simply, if a program can be judged to have a high degree of 
plausibility using the preceding criteria then it is an easy matter to discuss the different 
kinds of evaluation of the program that might be useful from implementation/service 
delivery to formative/improvement to summative/impact. Some of these may not be 
feasible given the degree of maturity of the program or the complexity of the effects and 
their consequences; nevertheless, their discussion is not difficult. 

5.9.4 Using the Criteria 

Judgments about plausibility will be made, such judgments will be influenced by the 
reasons they are being made as well as by those who make them. Competitive situations 
be they for funds or lead roles may find the criteria applied more stringently than for 
program development. Similarly, those who are involved in the program might apply the 
criteria differently than would a disinterested, third party. 

In the program design workshops to-date not much time or effort is devoted to the 
concept of plausibility. By the time the design team works through the senes of 
systematic, disciplined steps they have given their very best judgments as to what the 
form and some of the substance of the program should be. In addition, they are tired. 
They are not inclined to be critical of what they have done. Further, if time was sufficient 
and the Facilitator did his or her job, there should be nothing further to say. Plausibility 
should be redundant with all that preceded it. Plausibility was what it was all about! 



5-37 

34 



6. Incorporating Relevant Documents Into the Process 



If documents pertaining to prior, related programs, relevant research or evaluation results 
exist and are to be useful to the group their contents need to be part of their 
deliberations. Ideally, members of the design team will have sufficient familiarity with 
the documents to reflect upon the relevance of their content for the group. Short of this, 
arrangements need to be made by the administrative contact person to identify the 
documents and arrange for some of the group to review the document(s) between the first 
and second series of workshops and report back to the group. It is preferable but not 
always practical to have at least 2 members review the same document in order to allow 
for some possible differences in interpretation to emerge, if such exist. In the event that 
such a task is too time consuming for any of the group the administrative contact person 
will have to arrange for someone else to review the materials according to some 
specifications set down by the group and report back to them. Usually someone of the 
level of a graduate student or the person who does the interviews can perform this task 
fairly readily. Another alternative is to enlist the aid of colleagues of the group members 
thereby not only getting the work done but also maintaining the involvement of others who 
are important to the acceptance of the results of the group's efforts (see Russell, 1989). 

Very little has been done with the analysis of documents to-date. This is due in part to 
the^relative scarcity of such for the topics being considered. In addition, a rather narrow 
definition of "document" has been used. Russell, in her review of the 4-H program in 
Connecticut has used a much broader definition of "document" and has done what 
probably amounts to the most through and comprehensive "document" review to-date. 
In this process she and other staff members have reviewed: accountability reports; staff 
newsletters; minutes of staff meetings; 4-H participation type data; staff training and 
publications; funding sources-public and private; earlier mission and goal statements; 
staff involvement with volunteers; classroom based activities; prior program reviews, etc. 
The reader is referred to her report (Russell, 1 989) for details on an extremely imaginative 
and thoughtful approach to what can be done using a more comprehensive definition of 
"document." Also, as intended, this approach maintained the-- involvement of other 
colleagues of the design team members. 
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7. Identifying Stakeholders and Determining their Viewpoints 

This chapter introduces in more detail the concepts of stakeholder and stakeholder 
interviews. A sei of questions are adapted/developed for the interviews and individual 
stakeholders for different categories are identified. Interview procedures are developed 
and group analysis procedures are outlined including how the group arrives at a set of 
conclusions from them. Thematic observations are made which are included in 
subsequent steps. 

7.1 Stakeholder Identification and Questions 

In Chapter 2 we made a distinction between persons who could be considered 
stakeholders to an organization and those who could be considered stakeholders to a 
topical or thematic area. They are not necessarily the same persons. Figure 7.1 
attempts to illustrate these relationships wherein the area encompassed in the rectangle 
represents knowledge of the organization and the area encompassed in the circles 
represents knowledge of the topic. For example, a stakeholder to an organization is more 
likely to know something about the organization's involvement in a number of different 
topical areas (as represented by the rectangle). Conversely, a stakeholder to atopic may 
know a great deal about the topic but have partial (topic A), limited (topic C) or no 
knowledge (topic D) of the organization and its' involvement in the topic. Alternatively, a 
stakeholder to a topic may know a great deal about the organization as it is involved in- 
the topic but nothing more about either one ( topic B). 

In this section we are interested in stakeholders to a topical or thematic area. But how 
do we define stakeholder for these purposes and why do we seek information from them? 
A stakeholder is an individual (or group) who has a special interest in or influence over 
the topical area or program-to-be and who can provide information that will be useful for 
the development, implementation and evaluation of the anticjbated program*. Information 
is obtained from stakeholders in order to produce or shape a program so that it will be 
more "robust" and "impactful." "Robust" in the sense that it has the potential for survival 
throuah the involvement and support of parsons who are important for its future. 
"Impactful" in the sense that it will more likely produce trie desired changes with the 
intended audiences or target groups, by identifying differeijces in viewpoints about what 
is intended and even trying to resolve them at a later dare, if necessary. 

Early in the first session the concept of stakeholders is introduced and the group is asked 
to identify some general categories. This seems to help them work with the concept as 
well as to help map the "terrain" in which the program operates. Usually they can readily 
identify general categories. In fact, they usually identify too many and the categories 
have to be reduced in number later on. 



* See Benveniste (1989) and Majchrzak (1984) for other definitions of stakeholder. 
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Figure 7.1 Comparison of Organizational 
And Topical Stakeholders 




By the time the concept of stakeholders is revisited later in the first session, the 
workgroup has already done some modeling and has a better ic^ * of what the whole 
process is about. Hence, they are clearer in their own minds as to what the prpgram-to- 
be is that they are discussing. 

In revisiting the concept the group is first asked to develop a set of questions about the 
topic that they want to use with stakeholders. As a discussion point they are given an 
illustrative set of questions and asked to adapt them to their particular situation. Those 
questions given in Table 71 are a sample that have been used and refined based on 
experience gained through many of the case studies cited earlier. They are "not" 
intended to be adequate as is because they were developed for existing programs 
and not for new or never-before-done kinds of topics or programs. They may be 
useful with some modifications or they may be irrelevant and completely inappropriate. 

For example, in working on Import Substitution on Guam, Workman (1988) interviewed 
growers about their perceived barriers in producing certain commodities so as to better 
determine what the structure and content of the program should be. As a consequence 
the questions were very different from the sample ones. In general, the less past 
experience there is to build on the more likely it is that the questions will take on 
some kinds of needs assessment appearance, even though the questions may be 
more specific than the usual kinds of needs assessment questions (United Way, 
1982; Johnson, 1987; Kettner, et. al., 1990; McKillip, 1987). 
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It is instructive to review the questions and their intent (as indicated in the parentheses) 
in Table 7.1 before adapting them because they attempt to systematically elicit responses 
from stakeholders with regard to a particular extant program. [ We should recall that 
the interviewees never see these questions.] The first question is intended to serve 
as a basis for refreshing the interviewee's memory and to provide the interviewer(s) with 
a basis for understanding and interpreting later responses. The second question also 
serves as a refresher but allows the interviewees to reflect the organization or structure 
of their thinking with regard to the subject. Question 3 and 3a try to elicit views on 
current goals or others that might be pursued. Similar queries are made for target groups 
with questions 4 and 4a. Questions 5, 5a and 5b focus on perceived benefits (primary 
and secondary) or spinoffs and the things that Extension might do that they are not 
currently doing, to bring about these benefits. Question 6 is an increasingly important 
question for interagency and/or interdisciplinary work while 6a attempts to gauge the 
nature of needs that are not met by any of the agencies involved. Question 7 gauges 
stakeholder's perceptions of resource adequacy. Questions 8 and 9 try to elicit views on 
barriers, barrier reductions and intervening events while 10 tries to determine what it is 
they would like to know about the program, if anything. Eleven taps their thoughts about 
the future of the program while 1 2 is used as a vehicle for closing the interview. Question 
13 can be used as a means of "snowball" sampling to increase the number of 
stakeholders to be interviewed, if such is desired. Table 7.2 shows how these questions 
were modified to suit Iowa's Retail Trade Program, one for which there was a 
considerable amount of past experience. Table 7.3 shows how the Louisiana Water 
Quality workgroup modified these questions to a topic for which a prior program had not 
existed. 

In deciding how many stakeholders should be interviewed and which ones, a number of 
points need to be emphasized. First, this is not an exercise in statistical sampling but 
rather a judgmental process-there is no numerical quota for each category or overall 
(Brewer & Hunter, 1 989; Henry, 1 990; Patton, 1 990). However, for some categories (e.g. , 
Extension staff) we do try to build in variation by region, gender and ethnic background, 
if appropriate. (If not appropriate, it still doesn't hurt to keep- them in mind.) Alternates 
or back-up interviewees might aiso be planned for some categories. Second, one has 
to keep asking the question whether or not a potential interviewee both knows something 
useful about the topic and may be in a position of influence with respect to it. There may 
be a tendency to want to use the interviews as a device for educating interviewees about 
the topic. Such is not the purpose of the interview and if it is felt necessary to provide 
the interviewee with information about the topic then it is likely that the person is not a 
stakeholder. A general rule-of-thumb bom of experience is that "the greater the need 
to provide the interviewee with information about the topic the less likely it is that 
the person is a stakeholder". The exception of course, is that some information must 
be given in the contact letter and introductory statement-more will be said about that 
later. Finally, one of the considerations governing the number of interviews to be 
conducted is the cost and time required to conduct, transcribe and analyze them. A coot 
per interview figure is not used since many of the costs can be absorbed via existing 
resources (e.g., staffs secretaries doing transcription work, etc.). A small number, about 
30 interviews per topic is recommended but one can end up with 30 to 100. Pre-tests 
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Table 7.1 Illustrative Example Of Stakeholder Interview Questions For Program XYZ 

1. What has been your experience with the Extension XYZ Program (MEMORY REFRESHER 

of the University of ? & BASIS FOR 

UNDERSTANDING 

2. What are your overall views of Program XYZ in this State? LATER RESPONSES) 

3. What do you think Program XYZ is trying to accomplish? (GOALS) 

a. Are there other things you think it should be trying to accomplish? (OTHER GOALS) 

4. Who do you think is being served by this program? (TARGET GROUP) 

a. Are there others who you think should be served? (OTHER TARGET GROUPS) 

5. What benefits do you feel that people receive from their participation in this 
program? (BENEFITS) 

a. What other benefits do you think result from this program? (SECONDARY BENEFITS 

OR SPINOFFS) 

b. Are there other things Extension could do to bring about these 
benefits? (ADDITIONAL EFFORTS) 

6 How do you feel that Program XYZ operates in conjunction with those of other 
organizations serving ? (WORK WITH OTHER AGENCIES) 

a. Do you feel that there are needs that none of those groups are dealing 
with? (UNMET NEEDS) 

7. Do you think the resources of the Program are adequate? (RESOURCES) 

a. If yes, in what ways do you feel they are adequate? 

b. If no, what more do you feel is needed? 

8. Do you feel that there are difficulties or obstacles that staff &/or volunteers 
have to deal with in carrying out the program? (BARRIERS) 

a If so, how do you think that these difficulties can be dealt with? (BARRIER REDUCTIONS) 

9. In your view, are there any particular conditions or obstacles that make it difficult 
for clientele to benefit from their participation in this program? 

(BARRIERS or INTERVENING EVENTS) 

a. If so, what might be done about them? (BARRIER REDUCTIONS) 

10. If the program were to undergo further study or evaluation, are there questions you 
would like to see answered? (INFORMATION DESIRED) 

1 1 . In what ways do you think the program should change to meet future needs in this State? 

(FUTURE PERSPECTIVE) 

12. Do you have any other thoughts or ideas about this program that you would like to 
share with us? (CLOSE OUT) 

13. Are there others you can suggest that we talk with concerning Program XYZ? 
(Optional - to be used for expanding stakeholder interviews.) (OPTIONAL) 
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Table 7.2 Retail Trade Programs' Stakeholder Questions 

1 . What has been your experience with the Iowa State University Extension Retail Trade 
Program? 

2. What are your overall views of the Retail Trade Program? 

3. What do you think the Retail Trade Program(?) is trying to accomplish? 
a. Are there other things you think they should be trying to accomplish? 

4. Who do you think is being served by this program(s)? . 
a. Are there others who you think should be served? 

5. What benefits do you feel that people receive from their participation in this program(s)? 

6. What else should Extension do to bring about these benefits? These could be 
topics, delivery methods or other efforts. 

7. How do you feel that the Retail Trade program(s) operates in conjunction with those of 
other organizations ( such as Small Business Development Centers, community 
colleges, private consultants, & others)? 

a. Do you feel that there are needs that none of these groups are dealing with? 

8. Do you think the resources of the program(s) is adequate? 

a, If yes, in what ways do you feel they are adequate? 

b. If no, what more do you feel is needed? 

9. If the program were to undergo further study, are there questions you would like to see 
answered? 

10. Do you have any other thoughts about this program(s) that you would like to share with 
us? 

Thank you very much for your time and the information you shared with us? 
Adapted from Hammond.D. et al (1989) 
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Table 7.3 Stakeholder Interview Questions for Louisiana Water Quality Program Design 



1. In your capacity as * . or, in your association with * T in what ways is 

(title) (organization) 
** involved in water quality/quantity concerns? 
(organization) 

2. Are there water quality/quantity problems that you feel ** should deal with? 

(organization) 

3. What do you see as the most important concerns/issues affecting water 
quality/quantity over the next year? Over the next 5 to 10 years? 

4. What additional information does ** need to know to deal with your water 

(organization) 

quality/quantity concerns? 

5. What do you feel the goneral public needs to know about water quality/quantity? 

6. What do you feel the elected officials or regulatory agencies need to know about 
water quality/quantity? 

7. Who do you feel should be addressing the water quality/quantity problem? 

8. Are you familiar with the Louisiana Cooperative Extension Service county agent or 
home economist or 4-H agent? 

8.1 If yes, . 

(a) In what ways? 

(b) Do you think LCES should play a role in addressing water quality/quantity 
concerns/problems? 

(c) How? 

9. Do you have additional concerns/suggestions about water quality/quantity? 

Use title/position of the person in the organization/agency. 
** Use name of the organization/agency. 
Adapted from Verma, S. (1990). 

are useful and a small number should also be planned (e.g., 3-6). One way to keep the 
number of interviews down is to prioritize the categories and the interviewees within each 
category. In this manner, if time and money run out, the highest priority ones will be 
completed. 

Most of the interviews conducted to-date have been individual interviews conducted over 
the phone or in a face-to-face setting, usually by one person. However, for some 
purposes group interview techniques are more appropriate. For example, in Louisiana's 
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7.4 Louisiana's Leadership Development Program 
Design: Focus Group Interview Questions 



Opening Statement 

In a democratic society such as ours citizen participation through group activities and programs Is vary 
i*»rtait. And, for groups to function effectively we need the people who have the abilities and skills 
to act as leaders. 

For a nufcer of years the Louisiana Cooperative Extension Service has been providing training and 
educational experiences to help people learn the skills by which they could assise leadership roles 
in fart lies, organizations, and the broader coaaunlty. Soae of the leadership skills the 
Extension Service has tried to iepart are organizing groups, serving as officers, leading discussions, 
guiding decision aaking, teaching aid others. The Extension Service is reviewing these efforts to see how 
it should reorient aid adjust the leadership development effort to aeet the changing needs of our audiences 
as they prepare to f*ce the chal lenges of the 21st century. You were chosen to be a part of this grot* 
because of your knowledge and experience. 



Quest ioning Route 



1. To start off could you tell us soae of the leadership activities in which you have been involved? 

2. Did the Extension Service provide aty experiences that helped you in these leadership activities? 

3. Are there other leadership skills that could have benefited you in the past and rtiich you feel you will 
need In the future? 

4. Are there soae leadership skills that other people aay need so as to assuae leadership roles in society? 

5. In your view, who (or which organizations) should have the responsibility for leadership training 
in Louisiana? 

6. Do you have soaething else you would like to add? 
Cone I ud 1 no Statement 



Thart you for participating. We shal I share the results of this discussion with you. 
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^J c BEST COPY AVAILABLE 



Table 7.5 Stakeholder Contact Letter for Iowa's 

Retail Trade Program 



Dear (name):* 



Iowa State University Extension Service Is reviewing its retail trade programs in an effort to laprove 
our educational offerings. You nave bees Identified as an individual who could provide useful information 
regarding our Retail Trade Prograas. 

You will be contacted in a few days by (naae) to set up a tiae for a telephone interview. The interview 
will consist of 19 questions and takes approximately 15-2« ainutes. We will be tape recording the 
interview in order to accurately preserve your coaaents. However, the suaaary report will contain no 
cements attributable to any individual involved in this process. Your coaaents will be treated in 
strict confidence. 

The retail trade efforts of Iowa State University Extension have included the retail trade analysis, 
business aanageaent prograas, custoaer relations, new business start-up, and consumer iaage studies 
aaong other related prograas. The questions will deal with your knowledge and perception of how these 
prograas have been conducted and how they could be Improved. 

I nope that you will be willing to take 15 ainutes to assist us in this program review. If you have any 

questions or concerns, please contact (name) at (phone #). 

Thank you for your assistance. You can look forward to a cal I in a few days. 

Sincerely yours. 
Dean and Director 
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6 Stakeholder Contact Letter for Louisiana's 
Water Quality Program 



Dear (name):* 

The Louisiana Cooperative Extension Service (LCES) is the educational arm of the LSU Agricultural Center 
and conducts non-formal educational prograas In agriculture, home economics, 4-H youth and econoaic and 
community development to benefit Louisiana families. The LCES has identified water quality/quantity as an 
issue of wide public concern and is determining the need for educational programs in this area. You were 
identified by a working committee of LCES as a representative of your organization that could provide 
useful information and Insights regarding water quality/quantity concerns. 

You will be contacted In the near future to set up a time for a telephone Interview. The interview will 
consist of 9 questions concerning your ideas about water quality/quantity and will take approximately 
15 to 29 minutes, we will be tape recording the interview In order to accurately preserve your comments. 
However, the summary report will contain no comments attributable to any individual involved In this 
process. Your comments wi 1 1 be treated in strict confidence. 

Thmnk you for your participation in this effort. If you have any questions or concerns, please 

contact (name) at (phone #). You can look forward to a call In a few days. 

Sincerely, 



Y Ice-Chance I lor and Director 
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O" w Note : it is Important that the contact letter not specifically Identify the questions to be 
ERJC M |(ed nor include information which would provide answers to them. 
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Table 7.7 Stakeholder Categories for the HNDP Program 

CATEGORIES/NUMBER OF PERSONS 
NMC President - 1 

Board of Regents - experienced members - 3 (one each from Saipan, Tinian, Rota) 
Manager. Saipan Farmer's Market - 1 
Meager, Tinian Farmer's Market - 1 
Director, Department of Natural Resources - 1 
Senate Chairman on Health, Education & Welfare - 1 
House of Representatives Chairman on Appropriations - 1 
Growers - 3 (one each from Saipan, Tinian, Rota) 
Homemakers 3 (one each from Saipan, Tinian, Rota) 
Total - 15 



Table 7.8 Stakeholder Categories for the 
Retail Trade Program 

CATEGORIES/NUMBER OF PERSONS 

Iowa State University Extension Staff 

Adalnlstrator - 1 
Area Directors - 2 
Area Specialist - 1 
State Specialists - 2 
Owty Staff - 3 
Total - 9 

Outside Agencie s Knowledgeable of Retail Trade Programs 

Other Providers - 2 
State Agency - 1 
Congression al Staff - t 
Total - 4 

Local Community Leaders Viho Participated in Retail Trade Programs 



Total - 19 - from comities of varying size: 21W; 2908; 6390; 12,6W 



Grand Total - 32 
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Leadership Development program design, focus group interviews were conducted 
mmeoer %88; Morgan, 1988 Stewart & Shamdasami, 1990) because the wori<group 
e vZ he interviewees would better articulate their views if they couU share and rated 
upon their experiences with others (Verma, 1991). The questions they used for these 
purposes are given in Table 7.4. 

The initial letter contacting the stakeholder is a very critical document. See the samples 
in Table 7 5 for the Retail Trade Program ( for which there was a program) and Tab e £6 
or Si JantfJ Water Oiality design (for which there had not been a pnor program). The 
MlS^P«i to aJlow the interview and to tell them a ttte*«* 
why one wants to do so (viz., provide background on the purposes and detail). Howwer, 
Hhouldi Tot provide information that will serve as answers to the questions. 
qtaSdere need also to be assured of the anonymity of their responses and that 
SSSton^ *£^****> individuals will not be made, ft is felt that a high level 
^n^ such as the Director of Extension, lends importance and credibility to the 
2225? If the Interview to be conducted face-to-face, over the phone, in a group or 
SttS^^cSS letter should so state. If it is to be recorded verbatim then they 
should S told of this also. Some states have very stringent rules about conducting 
interviews, such as getting written permission beforehand. 

The desian team may choose to devote some time to adapting and/or drafting the contoct 
tettefoX ^ayToose to let some members of the group work with the admmist^rve 
contort 'person at a later time (usually the latter). At this point the workgroup can mm 
toe ^ SoUtakeholder categories, eliminate some and begin providmg names for others. 
See ^at^ZlTa^ll for i ustrative examples from the HNDP program, the Retort 
TmZ program and Louisiana's Water Qualfty effort. ">^^*™ 
should be provided to the administrative contact person while the > group sst.ll together 
to the extent possible, so as to accelerate the process and reduce the number of follow- 
ups that have to be made later. 

The ooenina statements of the interview are also critical irr setting the stage without 
Iravidtoc ^a?swere to toe questions (see the sample introductory statement in Appendix 

contact person). The 

hv a third neutral oartv-someone who is nd committed to the topic and who ooesnt 
have to have any s^dal topical expertise. Graduate students and even sometimes 
superlsec^ariycan do these very well if gfcen adequate supenjision and « 
However, special skills are required for focus group interviews (Kmeger^ V 388^ To-dae 
much use has been made of phone interviews conduded by a third party that are tape 
recoroe^vemattm wed verbatim and then these transcripts are used as the basic 
documents for analysis purposes. 
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7.2 Analyzing Stakeholder Viewpoints 

There is a great deal of analytic work that can be done with the interview results-usuaily 
far more than the team will ever have time to do while they are together. For purposes 
of the following discussion it is assumed that a typed transcript of each interview is 
available on a diskette that can be entered into a computer for analysis purposes as well 
as printed out. These transcripts are the basic analysis documents. It is also possible 
to distinguish between analytic work that the team does together and analytic work that 
is done prior to and perhaps also, subsequent to the their meeting(s). This latter work 
is usually done by someone (e.g., a graduate student) who is not a member of the team. 
Prior experience has shown that the work of the group and its comprehension of the 
results can be greatly enhanced if there has been a good deal of analysis done prior to 
the second session of the group. For the most part this work has been done on a 
computer using one of several content analysis programs (e.g., Gofer, Asksam, FYI 3000 
etc.). The results are usually organized by each question and by each category of 
stakeholder. They may be a "shredded" version of the interviewees' responses <viz., 
selected statements from a paragraph) or their complete response to the question. The 
advantage of organizing the results this way is that the individual respondent is not as 
readily identifiable (assuming of course that there are 2 or more persons in each category 
of stakeholder). Ratings may be made of the judged familiarity of the interviewee with . 
Extension generally, the subject-matter etc. and a great deal of analyses can be done if 
time and resources permit (see Mortvedt, 1990; 1991). 

Regardless of the form in which the results are presented to the group the basic idea is 
to have them feel that what was done was reasonable, that the summaries and 
conclusions they develop are accurate, and that the important inferences are made 
by them. On handing out the materials that the group will be working with, the following 
steps are usually followed: 

1 . They are reminded that the purpose of the exercise is to discern trench in 
viewpoints among stakeholders and stakeholder groups~and not to identify who a 
particular respondent might be. If the latter occurs such results should be kept within the 
group and not mentioned or discussed otherwise. 

2. The interview results are divided up so that at least 2 members of the team 
will read all of the results for a single category of stakeholder, discuss their observations 
with one another and reach agreement on what they have read. For all of the categories 
of stakeholders the work can be distributed fairly equally by having different members pair 
up (or triple up, etc.) with others, some taking a fewer categories that are more heavily 
represented, others taking more categories that are less heavily represented etc. 
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3. Once the members have been identified they are given a period of time 
(usually 1 1/2-3 1/2 hours) to read the results, discuss them and arrive at some very 
cryptic summary statements. 

4. The group is reconvened and a spokesperson for each analysis sub-group 
and stakeholder category narrates their results to the Facilitator(s) who puts them in a 
large matrix that has been taped on the wall. This matrix contains rows for the questions 
and columns for the stakeholder categories, as illustrated in Figure 7.2. 

Examples of such summaries from the Retail Trade Program and from Louisiana's 
Water Quality program design are given in Table 7.10 and 7.11, respectively. 

Once the matrix has been completed the group is asked to identify any themes or 
observations that occur to them by virtue of these results. This list of Themes can vary 
widely in its content and that's ok. It is meant to capture whatever thoughts they may 
have before moving on to the other topics. They may also request that further analysis, 
verifications etc., be made and included in the program design report. See Tables 7.12, 
7.13 and 7.14 for examples from the HNDP program, Retail Trade program and from 
Louisiana's Water Quality program design. 

The results of the stakeholder analyses represent the "pinnacle" of informed opinion with 
regard to the topic because they involve the viewpoints of a group that the design team- 
judged, on the basis of their experience, to be the most knowledgeable. Viewpoints of 
others would likely represent a "descent" from this pinnacle because they would not be 
as knowledgeable. If it were possible to select a random sample of all possible 
stakeholders, results frc.n interviewing them would not be nearly as informative. The 
magnitude of the discrepancy of the results for these two procedures would be a test of 
the expertise of the workgroup. The larger the di* ^pancy in favor of the judgmental 
sample, the more "expert" would the team members be. The smaller the discrepancy, the 
less "expert" they would be. Unless, of course, all possible stakeholders were equally 
, informed. Then, and only then, could the difference not be reflective of their 
knowledgeability for all would be equally well informed or equally poorly informed. The 
fact that the team would not know the extent to which stakeholders are informed however, 
would be a sad commentary on "their expertise". 
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Figure 7.2 
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Interview Results 

Stakeholder Categories 


Summary 
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ABC 
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6. 
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7.3 Summary of the Sequence of Steps Involved in Identifying Stakeholders and 
Determining their Viewpoints 



Figure 7.3 summarizes the sequence of steps involved in the determination of stakeholder 
viewpoints. One first has to decide whether the interview will be of an individual or group 
form. If the former then will it be face-to-face, over the phone or in some combination? 
These decisions will affect the nature and number of questions as well as the letter and 
contact procedures. Once contact has been made and the interviews conducted they 
must be transcribed and organized in such a manner as to facilitate analysis by the group 
as well as by anyone else who will be doing analytic work either in preparation for the 
group meeting or as a result of it. The analysis and cryptic summarization by the group 
results in a series of thematic observations that provide information for the modeling as 
well as for recommendations and next steps. 
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Table 7.12 Themes/Obser ations from Interviews for 

the Hi jP Program 

- HIXED VIEWS REGARDING THE FUTURE OF AGRICULTURE IN CNHI 

- ALL STAKEHOLDERS RECOGNIZED THEROLEpF LAW GRANT IN THE RJTURE OF AGRICULTURE IN CNMI AND HAD 
SPECIFIC IDEAS REGARDING WHAT THE ROLE SHOULD BE 

- SOME STAKEHOLDERS DO NOT HAVE A CLEAR IDEA OF THE FUNCTIONS OF LAND GRANT VIS-A-VIS 
THE DEPARTMENT OF NATURAL RESOURCES 

- MOST STAKEHOLDERS ARE NOT FAMILIAR WITH THE HNDP PROGRAM 

- STAKEHOLDERS ARE REQUESTING INFORMATION REGARDING WHAT THE PROGRAM IS OR ABOUT SPECIFIC ASPECTS OF IT 

- STAKEHOLDERS REQUEST THAT PROGRAM INFORMATION BE PUBLISHED AND USED IN CLIENT CONTACTS (BILINGUAL?) 

- ALL STAKEHOLDERS HAD SPECIFIC RECOMMENDATIONS REGARDING FUTURE PROJECTS/PROGRAMS 



Table 7.13 Themes/Observations from Interviews for the 

Retail Trade Program 

- RETAIL TRADE PROGRAMS ARE VIEWED POSITIVELY BY STAKEHOLDERS 

- TRADE ANALYSIS PROGRAM APPEARS TO BE THE STANDOUT 

- RETAIL PROGRAMS TEND TO BE ASSOCIATED WITH THE STATE SPECIALIST 

- OVERALL THE RETAIL TRADE PROGRAM LACKS A CLEAR IDENTITY 

- THE NEED FOR SERVICES OUTSTRIPS THE CURRENT CAPABILITY OF EXTENSION TO MEET THEM 

- STAKEHOLDERS FEEL THAT BUSINESSES AND COMMUNITIES ARE BEING SERVED 

- STAKEHOLDERS FEEL THAT THEY RECEIVE INFORMATION FROM EXTENSION THAT IS BOTH UNIQUE AND CREDIBLE 

- EXTENSION ADMINISTRATION AND OTHER STAFF SEEM TO HAVE A LIMITED VIEW OF OR KNOWLEDGE REGARDING 
THE PROGRAM 

- STAKBCLDERSWDULD LIKETO ^ 

(E.G NEWSLETTERS, ELECTRONIC MEDIA, ETC.) AVAILABLE 

- A LINK NEEDS TO BE MADE BETWEEN THE PROGRAM AND ECONOMIC DEVELOPMENT 

- STAKEHOLDERS FEEL THAT THE PROGRAM IS NOT ADEQUATELY MARKETED 

- GENERALLY. STAKEHOLDERS DID NOT FEEL THAT fXTCNS^ DUPLICATED- OFFERINGS OF OTHER AGENCIES. 
HOWEVER, OTHER AGENCY STAFF WOULD LIKE TO SEE MORE COOPERATION 

- WHERE FURTHER NEEDS ARE PERCEIVED, THEY TEND TO BE IN THE CUSTOMER RELATIONS/CONSUMER IMAGE AREA . 
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Table 7.14 Thematic Observations Based Upon the Interview 
Summaries for Water Quality in Louisiana 

- BOTH QUANTITY AND QUALITY ARE ISSUES 

- MOST STAKEHOLDERS ARE COMMITTED TO RESOLVING THESE ISSUES, BUT THERE ARE DIFFERENT PERSPECTIVES ON 
THE NATURE OF THE PROBLEM 

- AGRICULTURE IS HEAVILY INVOLVED IN WATER QUALITY ISSUES 

- CONSENSUS THAT THERE ARE CONTAMINANTS TO BE CLEANED AND A LEVEL OF NON-CONTAMINATION TO BE PROTECTED, 
BUTLACK OF AGREEMENT ON WHAT THE LEVELS ARE/SHOULD BE 

- NEED FOR BASELINE DATA AND RESEARCH INTO SOURCES AND TYPES OF SOLUTIONS 

- NEED FOR EDUCATION TO CREATE GREATER PUBLIC AWARENESS AND KNOWLEDGE 

- POSITIVE ACTIONS ARE BEING TAKEN REGARDING WATER QUALITY/QUANTITY CONCERNS 

- MANY AGENCIES ARE AND NEED TO BE INVOLVED; AT THE SAME TIME THERE IS A NEED FOR BETTER COOPERATION 
AND COORDINATION OF AGENCY EFFORTS 

- LCES SHOULD PLAY A KEY ROLE IN EDUCATION AND INFORMATION DISSEMINATION AND A FACILITATIVE 
ROLE WITH OTHER AGENCIES 
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Figure 7.3 Summary of Sequence of Steps Involved in 
Identifying Stakeholders & Their Viewpoints 
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8. Developing Conclusions, Recommendations & Suggesting Next Steps 

The final step of the program design process is for the group to identify a set of 
conclusions, provide a list of recommendations and indicate, if at all possible, what they 
think the next steps should be. Examples for the HNDP Program, Retail Trade Program 
and for Water Quality are given in Tables 8.1, 8.2 and 8.3 respectively. There is one 
recommendation that is an absolute must and is made by the facilitator to start off the 
list. It is: Provide feedback to those interviewed. This is an important point that must 
not be overlooked. Busy people took 20 to 30 minutes of their time to share their thoughts 
about the topic. The organization is obligated to provide them with some note of thanks 
for their comments, expressing that the comments were useful and indicating, in a general 
way, how they were used. To fail to do so would be bad procedure as well as bad public 
relations. One reaches out to stakeholders to involve them in a constructive manner and 
often on more than one occasion. [ Usually the State contact person will see to it ihat this 
gets done.] 

It is very common for stakeholders to know less about a topic or program than the design 
team members thought they did. Consequently, the group may feel that some efforts must 
be put forth to create an awareness of the organization &/or its involvement in the topic. 

Some ideas may occur to the group about new programming topics or efforts in which the 
organization might be involved. If so, these should certainly be put on the list. 

If one recommendation is that the resultant design be carried into the developmental 
stage then this usually has to be preceded by an executive decision involving the 
commitment of resources. However, the design may on occasion bypass the 
developmental stage and move right into the implementation phase. This is likely to occur 
for a topic in which there is a great deal of past experience or perhaps even a prior 
program and for which the changes are of a policy nature rather than structural or 
resource based. For example, in Louisiana's 4-H program design (Richard.R. & E. 
Johnson) one of the main recommendations involved how advisory groups should be 
constituted. If the policy decision were made to change the way they were constituted 
then agents would need some additional training to work with new audiences. However, 
the overall structure of the program would remain the same. 

If at all possible, an executive briefing should be given to the administrator &/or council 
by the design team. Usually the team will elect one or more spokespersons to make the 
presentation in their presence.! The facilitator(s) should not serve as such since the 
product is the design team's and ownership should be manifested through their 
involvement.] This serves a twofold purpose of giving recognition to the team members 
for their efforts while simultaneously making an impression on administration as to the 
seriousness and thoroughness of their efforts. 

The steps involved in this sequence are summarized in Figure 8.1. 
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Table 8.1 Conclusions ♦Recommendations 
& Next Steps for the HNDP Program 



- PROVIDE FREEDBACK TO INTERVIEWEES 

- REEXAMINE PROGRAM DELIVERY MECHANISMS & PUBLICATIONS 

- A BETTER UNDERSTANDING JifcEDS TO BE DEVELOPED BY STAKEHOLDERS OF THE 
ROLES OF LAND GRANT VIS-A-VIS THE DEPARTMENT OF NATURAL RESOURCES * 

PERHAPS OTHER AGENCIES TOO 

- WHAT WILL THE ENTRY POINTS BE FOR THE HNDP PROGRAM? 

- NEED TO IMPROVE THE PUBLIC RECOGNITION OF LAND GRANT & IT'S PROGRAMS 

INCLUDING HNDP 

- WHAT NEEDS TO BE DONE TO GET THE HNDP PROGRAM UP & RUNNING? 
- CONSUMERS-RET A-I LERSr PRODUCERS ? 

- USE PILOT AREA FOR INITIATING HNDP PROGRAM 

- THE PRIORITIES FOR ECONOMIC DEVELOPMENT IN CNN I ARE NOT CLEAR- 
THERE IS A NEED TO PROMOTE THE IDEA OF TOURISM-RELATED AGRICULTURE 



Table 8.2 Conclusions & Recommendations 
for the Retail Trade Program 

- A LOT OF POSITIVE FEEDBACK FROM VERY FEW RESOURCES 

- EXTENSION HAS A COMPARATIVE ADVANTAGE IN PROVIDING RESEARCH BASED 
INFORMATION REGARDING OTHER PROVIDERS 

- OPPORTUNITIES EXIST FOR EXPANDING THE PROGRAM 
IF RESOURCES WERE AVAILABLE: 

- SMALL BUSINESS CREDIT SCHOOL FOR BANKERS 

- FINANCIAL & OPERATING STANDARDS FOR SMALL BUSINESSES 

- DEVELOP A VIDEO TAPE FOR EFFECTIVE MARKETING FOR RETAILERS 

- PACKAGING MARKET POTENTIAL MATERIALS FOR SMALL BUSINESSES 

- NEED TO GIVE FEEDBACK TO THOSE INTERVIEWED 
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Table 8.3 Conclusions, Recommendations 
& Next Steps for LA* s Water Quality 

- PROVIDE FEEDBACK TO INTERVIEWEES 

- LCES SHOULD DEVELOP AN IDENTIFIABLE WATER QUALITY PROGRAM ALONG THE 

LINES INDICATED IN THIS REPORT & CONSIDER AS NEXT STEPS t 

- BRIEF LCES WATER QUALITY WORKING GROUP REGARDING THESE RESULTS 

- LCES WATER QUALITY WORKING GROUP SHOULD: 

I. PRIORITIZE AUDIENCES & OBJECTIVES 

II. PREPARE PROPOSALS FOR RESOURCES 

- OBTAIN RESOURCES FROM LCES ADMINISTRATION 

- FORM TASK GROUPS TO DO DEVELOPMENTAL WORK ON PRIORITIZED OBJECTIVES 

- COMPLETE THE ABOVE STEPS IN TIME TO INCORPORATE INTO THE 

NEW PLAN OF WORK CYCLE 

- INITIATE A QUARTERLY NEWSLETTER REGARDING WATER QUALITY EFFORTS IN LA 

- CONTINUE IN-HOUSE COMMUNICATION RE LCES ACTIVITIES IN WATER QUALITY 
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Figure 8.1 Summary of the Sequence of Steps Involved 
Developing Conclusions, Making Recommendations and 

Suggesting Next Steps 
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9. An Illustrative Model for Extension Programming 

This chapter presents an illustrative model based upon a synthesis of all of the program 
design/evaluability assessment work with which the author has been associated in the 
Cooperative Extension System to date (almost 3 dozen in total). This model can be used 
as an instructional and programming guide. However, it should never be used as a 
substitute for the design team's own best thinking. Figure 9.1 presents the illustrative 
Program Logic Model based upon this synthesis while Tables 9.1 through 9.13 contain 
the main events. Tables 9.1 1 and 9.1 2 pertain to the generic KASA and B/P entries from 
Figure 5.3. They also contain generic sources of evidence (indicators) that could be used 
to gauge their occurrence. Resources have not been specified since they vary so widely 
depending on the scope, structure and content of the program under consideration. As 
noted in Chapter 5, resource categories that have been used and then converted to dollar 
equivalents are: Professional Staff time (in FTE's) of County and State staff and for 
Specialists; Para-professionals; Volunteers; and, Secretarial Support. Others are: Travel; 
Equipment/Materials (Acquire/Purchase &/or Develop); and, Overhead. Figure 9.2 
presents the illustrative Expanded Program Logic Model with small circles used to depict 
the occurrence of Barriers or Intervening Events. The numerical entries in each circle give 
the number of the table in which they can be found (Tables 9.14 through 9.31). Figure 
9.3 presents the illustrative Program Logic Model again, this time using small circles and 
dashed lines to depict the occurrence of Spin-offs.The numerical entries indicate the 
Table in which they can be found (Table 9.32). 

On examination of these figures and tables the reader may observe the following: (1) 
indicators for activities tend to be primarily administrative documents that flow from the 
ongoing program while those for effects/consequences often involve some special effort 
to obtain; (2) effects indicators are often a mixture, of sources (e.g staff, other observers, 
etc.) and methods (e.g. survey, observations, feedback) while activity indicators are 
primarily source documents or files; (3) barriers are plentiful and this is especially so for 
KASA and for B/P changes (Tables 9.29 and 9.30, respectively)[Many of these are of a 
general nature not constrained to the specifics of a particular topic or technology.]; (4) 
spin-offs are also numerous and especially so for events that involve clientele or 
staff/volunteers. 

In subsequent chapters we regard the illustrative model as our program design and 
discuss how it might be dealt with in the developmental and implementation phases or 
cycles. 
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Table 9.1 . Activities and Indicators for Main Event 1 : 

Identity and Prioritize Problems 



IDENTIFY 
& 

PRIORITIZE 
PROBLEMS 

1 



ACTIVITIES 

• Key influentials/advisory councils 
identify, articulate and prioritize 
problems/need(s) with the assistance 
of Extension staff. 

• Extension gathers information from 
other sources concerning problem(s)/ 
need(s) (e.g., surveys, informal 
communications, observations, 
secondary data analysis, planning 
committees). 

• Extension compiles aggregate 
information regarding problem(s)/ 
need(s). 

• Extension and advisory committees 
interpret results and implications 

of preceding. 



Resources: 



INDICATORS 

County activity reports 
Council minutes 



Data on file 



Analysis on file 
Plan of work 



Analysis on file 
Plan of work 



12S 



Table 9.2. Activities and Indicators for Main Event 2: Define 

Extension's Role and Nature of Commitment 



DEFINE 
EXTENSION'S 

ROLE & 
NATURE OF 
COMMITMENT 



ACTIVITIES 

Extension staff review: (1) Extension 
mission; (2) Roles of other agencies; 
and, (3) Identified problem(s)/need(s). 

Extension staff use information (1-3) 
to identify those problems/needs to 
be dealt with. 

Extension staff describe the parameters 
of what is needed and determine the 
appropriate mix of education versus 
other services. 

Extension staff determine what is 
required/mandated and prepare plan. 

Extension administration reviews and 
sanctions proposed plan. 



INDICATORS 

Minutes of meetings 

Minutes of meetings 
Minutes of meetings 



Plan on File 
Plan of work 



Resources: 
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Table 9.3. 



Activities and Indicators for Main Event 3: 
Assess Needed and Available Resources 



ASSESS 
NEEDED & 
AVAILABLE 
RESOURCES 



ACTIVITIES INDICATORS 

• Extension staff identify: • Activity reports on file 

Campus-field relationships 
Other agencies-field relationships 
Inventory field staff capacity 
and capability. 

• Extension staff determine availability, • Activity reports on file 
accessibility and affordability of 

other resources (e.g., private 
industry, government, voluntary 
organizations, grants and donor 
agencies, etc.) 

• Inventory available materials, review • Activity reports on file 
for adequacy, accuracy and adaptability, • Inventory on file 
and estimate developmental needs. 

• Identify team members to design and • Memo from director to 
develop program. team members 



Resources: 
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Table 9.4. Activities and Indicators for Main Event 4: 

Acquire Resources, Design and Develop Program 



ACQUIRE 
RESOURCES, 
DESIGN & 
DEVELOP 
PROGRAM 



ACTIVIT IES 

Extension provides own funds and/or 
obtains other funds (e.g., grants, 
donations, etc.) to support start-up 
activities. 

Convene design team and determine 
design specifications including: 
•• Target audiences, desired results 

and consequences 
•• Programmatic steps to be taken 
•• Problems that may occur and how 

to deal with them (if possible) 
•• Unplanned occurrences 
•• Developmental tasks and needed 

staff identified. 

Administrative approval of design, 
developmental tasks and staff 
assignments. 

Convene development team and initiate 
tasks: 

•• Specify objectives and conduct materials 
review, subject-matter materials. 



INDICATORS 

Memo authorizing funds. 



Minutes/products of 
meetings. 

Recommendations to 
administration. 



Memo of approval and 
assignments. 



Minutes and products 

of meetings and consultations. 
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(Table 9.4. continued) 



•• Acquire and/or adapt/develop expertise. 
•• Tailor materials/expertise to meet 

special needs of target audiences 

(e.g., level of comprehension, time, 

availability, preferred delivery 

mechanisms, etc.). 
•• Conduct pilot tests and revise 

accordingly (as applicable). 

•• Develop plan for acquiring and/or 
training staff. 

•• Develop promotional plan and materials. 

•• Develop plan for networks/coalitions 
and establish contacts/relationships. 

•• Develop plan for ascertaining and 
communicating program accomplishments. 

•• Ascertain initial sites for implementation. 

Obtain administrative approval for 
implementation. 



Products and memos on file 
(e.g., focus group interview 
results). 



Products and memos on file. 

Products and memos on file. 

Products and memos on file. 
Products and memos on file. 

Products and memos on file. 

Products and memos on file. 
Memo of authorization. 



Resources: 
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Table 9.5. Activities and Indicators for Main Event 5: 

Involve Researchers 



INVOLVE 
RESEARCHERS 



ACTIVITIES 

• Subject-matter researchers participate 
in design and development in order to 
bring in their knowledge and perspectives 
and as a means of identifying needed 
research. 



INDICATORS 

• Participation in design and 
development meetings. 

• Individual consultations with 
development team members. 

• Research proposals prepared. 

• Research proposals approved. 



RESOURCES: 
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Table 9.6. Activities and Indicators for Main Event 6: 

Acquire and/or Train Staff and Volunteers 



ACQUIRE 
&/0R 
TRAIN STAFF 
& 

VOLUNTEERS 



ACTIVITIES 

Obtain applications or expressions 
of interest. 

Determine best qualified through screening 
of applications/expressions and interviews. 



INDICATORS 

Applications/expressions 
obtained and list on file. 

Interviews conducted and 
best qualified determined; 
list on file. 



Make selections and offer 
invitations/contracts. 

Provide training and/or orientation. 



Invitations and agreements 
made and on file. 

Activity reports containing 
information on who, what and 
when. 



RESOURCES: 



ERIC 
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Table 9.7. 



Activities and indicators for Main Event 7: 
Create Awareness and Promote Program 



CREATE 
AWARENESS 
& 

PROMOTE 
PROGRAM 



ACTIVITIES 

Make presentations to Extension staff 
at district meetings and in-service 
training sessions. 

Meet with other state agencies 
involved to inform them about program. 

Communicate availability and nature 
of services to targeted audiences, 
administration and select stakeholder(s) 
(e.g., presentations to civic organizations, 
elected officials, interested/concerned 
groups, etc.). 

Make media releases and disseminate 
promotional materials. 

Conduct promotional and recognition 
events. 



INDICATORS 

Minutes of meetings and activity 
reports. 



Minutes of meetings. 



Activity reports. 



Media releases and promotional 
materials on file. 

Activity reports. 



RESOURCES: 



ERIC 
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Table 9.8. Activities and indicators for Main Event 8: 

Initiate Networks/Collaborations 



INITIATE 
NETWORKS/ 
COLLABORATIONS 

8 



ACTIVITIES INDICATORS 



• Contact groups, agencies or • Activity reports, 
organizations to enlist their 

cooperation in program and 
obtain commitment. 

• Negotiate the relative roles • Minutes of meetings and 
and responsibilities of each memoranda of understanding, 
with respect to one another 

including staff, funds, etc. 

• Identify persons and/or products ■ Memoranda on file, 
involved, the nature of and 

schedule for their involvement. 



RESOURCES: 
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Table 9.9. 



Activities and Indicators for Main Event 9: 
Provide Educational Experiences 



PROVIDE 
EDUCATIONAL 
EXPERIENCES 



ACTIVITIES INDICATORS 



• Coordinator (e.g., agent) manages • Activity reports and 

program delivery team(s) (e.g., other minutes of meetings. 

Extension professional staff, 
para-professionals, volunteer leaders, 
school personnel, community leaders, 
policy-makers, etc. as appropriate) 
who perform the following kinds of 
activities: 

Obtain endorsement and/or 

co-sponsorship (as appropriate) 

from community groups. 

Schedule dates for meetings/ 

workshops/clinics regarding 

convenience, geographic locale 

and make facilitative arrangements 

(e.g., time, speaker, place, 

equipment.) 

••• Make conferencing or other media 

arrangements as applicable. 
Recruit participants. 
Provide ongoing assistance in 
recruiting and registration process. 



ERIC 
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(Table 9.9. continued) 



. Program delivery team(s) provide(s) • Activity Reports, 

educational experiences for target 
audiences including (as appropriate): 

Conduct meetings, workshops, 

consultations (one-on-one and 

group, media, etc.). 

Provide literature or other 

educational materials. 

Make referrals to other sources 

of assistance. 

Coordinate and communicate activities 
with staff and cooperating agencies. 



• Extension staff determine: 

Efficacy of collaborative and • Accomplishment information 

promotional efforts. on fi|e - 

Efficacy of delivery team(s) effort(s). • Accomplishment report 

developed and results 
disseminated (on file). 

Ascertain and communicate program 
accomplishments. 



RESOURCES: 
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Table 9.10. Activities and Indicators for Main Event 10: 

Involve Researchers 



INVOLVE 
RESEARCHERS 

10 



ACTIVITIES INDICATORS 



• Subject-matter researchers conduct 
needed research and infuse results 
into the program. 



RESOURCES: 



• Research reports and 
recommendations. 



ERIC 
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Table 9.11. 



Effects and Indicators for Main Event 11: 
Changes in Clientele: Knowledge, Attitudes, 
Skills and Aspirations 



CHANGES IN 
CLIENTELE: 
KNOWLEDGE, 
ATTITUDES, SKILLS, 
& ASPIRATIONS 

11 



EFFECTS 

Knowledge 

Attitudes 



Skills 



Aspirations 



INDICATORS 

(As appropriate to the structure 
of the program and the particular 
target audience). 

• Participation in meetings. 

• Direct and/or indirect feedback 
from participants and informed 
observers. 

• Staff observations. 

• Extension or other agency 
reports. 

• Legislation. 

• Follow-up survey(s). 



9 

ERIC 
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Table 9.12. 



Effects and Indicator- 'or Main Event 12: 
Changes in Clientele: Behaviors/Practices 



CHANGES 
IN CLIENTELE: 
BEHAVIOR/PRACTICE 
CHANGES 

12 



EFFECTS 



INDICATORS 



Behaviors/practices changed 
(including decision to adopt 
new practice(s)). 



Same as Main Event 1 1 
(see Table 1 1 ). 



ERIC 
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Table 9.13. Consequences and Indicators for Main Event 13: 

Clientele, Community and Societal Consequences 



CLIENTELE 
COMMUNITY 
& 

SOCIETAL 
CONSEQUENCES 



13 



CONSEQUENCES 



CLIENTELE BENEFITS 

• Personal conditions improve through 
adoption of recommended practices 
(e.g., efficiency and effectiveness 
of operations, nutritional status and 
and health, quality of family life, 
environment, financial status, 
self-esteem, outlook on life, etc).. 



INDICATORS 



• Increase in sales and/or profits. 

• Lower incidence of pathologies. 

• Lower incidence of child/spouse 
abuse, dropout, substance abuse, 
juvenile crime. 

• Less contaminants in water and 
food supply. 

• Restructured and improved 
management of finances. 

• Enhanced sense of personal 
well-being. 



COMMUNITY BENEFITS 



Community conditions improve through 
the cumulative effects of clientele 
benefits. 



Increase in number and quality 
of community leaders; more 
active citizenry; more 
empowered community; 
enhanced economic viability; 
enhanced sense of community. 
Improved local networking; more 
opportunities and services, less 
out-migration. 



ERIC 
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(Table 9.13. continued) 



SOCIETAL BENEFITS 

• Societal conditions improve through the • Improved living resources 
cumulative effects of clientele and (e.g., ground water and lakes), 

community benefits (e.g., environment, • Greater satisfaction with way 
quality of life). of life. 
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Table 9.14. 



Barriers and Reductions for Main Events 1 and 2 



IDENTIFY 
S 

PRIORITIZE 
PROBLEMS 



T BARRIER 1 



| BARRIER I 
I BARRIER I 
=tf BARRIER I 
I BARRIER..J 



DEFINE 
EXTENSION'S 

ROLE 4 
NATURE OF 
COMMITMENT 



BARRIER 
REDUCTIONS 



BARRIERS 

inadequate problem identification 
process. 

Belief that problem or need cannot 
be redressed. 

Perception of lack of receptiveness 
on part of potential target audience(s). 
Uncertainty of importance of 
problem/need. 



Resistance to change in general. 



Problem/need not a priority in 
county/state. 

No impetus from administration. 

Extension role not clear to 
staff/agents. 

Agent/staff time constraints. 
Lack of program development 
experience in program/topical 
area. 



BARRIER REDUCTIONS 

Diversify sources of 
information/informants/ 
advisors. 
Test the idea. 

Target special needs. 

Identify emerging problems/ 
needs earlier, improve 
communications, coordination, 
prioritization within Extension 
and with cooperating 
agencies. 

Reduce fear of change 

through training, staff 

diversification. 

Train councils/committees 

and agents. 

Provide administrative 

direction. 

Provide in-service training. 

Plan far enough ahead. 
Allow time for experience 
to occur. 



ERIC 
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Table 9.15. 



Barriers and Reductions for Main Events 1 and 3 



IDENTIFY 
& 

PRIORITIZE 
PROBLEMS 



f BARRIER 1 



| BARRIER I 

| BARRIER I 

=tf BARRIER I 

L BARRIER J 



ASSESS 
NEEDED 
& 

AVAILABLE 
RESOURCES 



BARRIER 
REDUCTIONS 



BARRIERS 



BARRIER REDUCTIONS 



Inadequate problem identification 
process. 

Belief that problem or need cannot 
be redressed. 

Perception of lack of receptiveness 
on part of potential target audience(s). 
Uncertainty of importance of 
problem/need. 



Resistance to change in general. 



Problem/need not a priority in 
county/state. 

No impetus from administration. 

Extension role not clear to 
staff/agents. 

Agent/staff time constraints. 
Lack of program development 
experience in program/topical 
area. 



• Diversify sources of 
information/informants/ 
advisors. 

• Test the idea. 

• Target special needs. 

• Identify emerging problems/ 
needs earlier, improve 
communications, coordination, 
prioritization within Extension 
and with cooperating 
agencies. 

• Reduce fear of change 
through training, staff 
diversification. 

• Train councils/committees 
and agents. 

• Provide administrative 
direction. 

• Provide in-service training. 

• Plan far enough ahead. 

• Allow time for experience 
to occur. 



9 

ERIC 
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Table 9.16. 



Barriers and Reductions for Main Events 2 and 3 



DEFINE 
EXTENSION'S 

ROLE & 
NATURE OF 
COMMITMENT 

2 


T BARRIER 1 


ASSESS 
NEEDED 
& 

AVAILABLE 
RESOURCES 

3 


I BARRIER 1 
| BARRIER 1 

— N BARRIER 1 

L BARRIER J 



LJ 

BARRIER 
REDUCTIONS 



BARRIERS 

• Uncertainty of funding. 
Extension staff and budget 
constraints. 

• Public perception that Extension 
does not have a role in the 
problem/topical area. 

• Lack of research base. 



Insufficient internal program 
communication. 
• Disciplinary and constituent 
orientation. 

Insufficient level of commitment 
to problem/topical area by 
decisionmakers. 



BARRIER REDUCTIONS 

• Obtain temporary funding. 

• Develop/seek other resources; 
use innovative staffing patterns 
and program delivery methods. 

• Market programs better. 



• Do high priority applied 
Extension research; communicate 
research needs to researchers. 

• Prepare timely program 
communication material in advance. 

• Move toward basic academic training 
in relevant areas or training on the 
job RE: relevant areas. 

• Continue communication 
RE: problem/topic/needs. 
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Table 9.17. 



Barriers and Reductions for Main Events 2 and 4 



DEFINE 
EXTENSION'S 

ROLE & 
NATURE OF 
COMMITMENT 



T BARRIER 1 



I BARRIER t 
| BARRIFR | 
=M BARRIER I 
L BARRIER J 



BARRIER 
REDUCTIONS 



ACQUIRE 
RESOURCES 
DESIGN 

DEVELOP 
PROGRAM 



BARRIERS 

Public perception of Extension. 
Insufficient level of commitment 
to problem/topical area by 
decisionmakers. 
Extension staff and budget 
constraints*. 



• Lack of research base*. 



• Limited knowledge of 
resource base*. 

• Insufficient internal program 
communication. 

• Disciplinary and constituent 
orientation. 

Lack of program development 
experience in problem/topical 
area*. 



BARRIER REDUCTIONS 

Market programs better. 
Continue communication re: 
problem/topical area. 

Seek advice regarding availability 
use innovative staffing patterns 
and delivery methods; seek more 
resources or outside funding. 
Do high priority applied Extension 
research; communicate research 
needs; maintain Strong relationships 
with regional development centers. 
Provide training; develop resources 
and directories; make referrals; 
brainstorm. 

Prepare timely program 
communication materials in advance. 
Move toward basic academic training 
in relevant areas or training on the 
job re: relevant areas. 
Allow time for experience to occur. 
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Table 9.18. 



Barriers and Reductions for Main Events 3 and 4 



ASSESS ' 
NEEDED 
& 

AVAILABLE 
RESOURCES 

3 


r BARRIER 1 ( 


ACQUIRE 
RESOURCES 
DESIGN 
& 

DEVELOP 
PROGRAM 4 


| BARRIER 1 
I BARRIER 1 
;{ BARRIER 1 
L BARRIER J 



c 



BARRIER 
REDUCTIONS 



BARRIERS 



BARRIER REDUCTIONS 



See Starred Items Under 2 and 4 in Table 9.17. 
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Table 9.19. Barriers and Reductions for Main Events 4 and 5 




ACQUIRE 
RESOURCES 
DESIGN 
& 

DEVELOP 
PROGRAM 



BARRIERS 

Problem/topic of little 
interest or low priority 
for researchers. 



Conflict in orientations to 
time of Extension versus 
research. 



Different criteria and standards 
of performance for Extension 
vs. research. 



BARRIER REDUCTIONS 

Increase level of interest/ 
priority through joint 
appointments and/or 
discretionary Extension 
funding. 

Plan for lead times; 
involve researchers 
earlier; plan for use of 
preliminary/interim results. 

Seek joint appointments 
and composite standards 
for pay, promotion and 
tenure. 
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Table 9.20. Barriers and Reductions for Main Events 4 and 6 



ACOUIRE 
RESOURCES 
DESIGN 
& 

DEVELOP 
PROGRAM 



T BARRIER 1 



I BARRIER I 
I BARRIER I 
=tf BARRIER I 
L BARRIER J 



c 



BARRIER 
REDUCTIONS 



ACOUIRE 
&/OR 
TRAIN 
STAFF 
& 

VOLUNTEERS 



BARRIERS 

Uncertainty of funding. 
Unavailability of "staff/ 
volunteers. 



Lack of means of accessing 
faculty expertise and interest. 
Lack of district/county staff 
"buy-in." 

County staff lack ability and 
interest in promoting program. 
Clientele and agents limiting 
stereotype of agent's and 
Extension's role. 



Inappropriate selection/ 
assignment of staff/volunteers. 
Lack of qualified trainers. 



BARRIER REDUCTIONS 

Seek other sources. 
Determine reasons and resolve 
(e.g., overload compensations, 
flexible scheduling, network with 
other knowledgeable people, 
develop volunteer cadre, recruit 
volunteers from new sources, 
etc.). 

Develop catalog or on-line data 
base. 

Market to county and district 
staff.. 

Tailoring program to local nesds 
with agent involvement. 
Provide personal contact 
and training, success 
experiences, communication with 
other staff; provide training and 
support in developing recruitment 
techniques, clientele testimonials; 
increase emphasis on national 
initiatives, establish programming 
units. 

Reassign responsibilities. 



Determine reasons and resolve; 
invite applicants; train trainers. 



ERIC 
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Table 9.21 . 



Barriers and Reductions for Main Events 4 and 7 



ACQUIRE 
RESOURCES 
DESIGN 
& 

DEVELOP 
PROGRAM 



f BARRIER 1 



| BARRIER I 
I BARRIER I 
=tf BARRIER I 
L BARRIER J 



BARRIER 
REDUCTIONS 



CREATE 
AWARENESS 
& 

PROMOTE 
PROGRAM 



BARRIERS 

Lack of funding at key points 
in time. 

Unrealistic time lines. 
Staff/volunteer overcommitment. 
Lack of access to media. 

Lack of promotional expertise. 

Failure to recognize need for 
promotional efforts. 



BARRIER REDUCTIONS 

Seek other sources/ 
resources; reduce efforts. 
Adjust time lines, 
overtime, improve 
planning, reduce offerings. 
Delegate, provide 
overtime, reassign 
responsibilities. 
Use alternative marketing 
strategies (e.g., direct 
mail); develop relationship 
with media. 
Provide training, gain 
access to expertise. 
Provide training. 



9 
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Table 9.22. 



Barriers and Reductions for Main Events 4 and 8 



ACQUIRE 
RESOURCES 
DESIGN 
& 

DEVELOP 
PROGRAM 



T BARRIER 1 



| BARRIER I 
| BARRIER | 
=tf BARRIER | 
L BARRIER J 



BARRIER 
REDUCTIONS 



INITIATE 
NETWORKS/ 
COLLABORAT- 
IONS 



8 



BARRIERS 

Lack of liaison to establish 
contacts. 

Resistance of other agencies/ 
organizations to collaborate in 
what they regard as "their turf." 
Bureaucratic inertia. 



Program is low priority for 
other agencies/organizations. 



BARRIER REDUCTIONS 

Create Liaison. 



Define the niches for 
eac~ nd the necessity 
for c Donation. 
Seep, assure points/ 
persons to expedite. 
Seek higher priority 
endorsements by high 
level administrators. 
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Table 9.23. 



Barriers and Reductions for Main Events 6 and 7 



ACQUIRE 
&/0R 
TRAIN 
STAFF 
& 

VOLUNTEERS 



T BARRIER 1 




| BARRIER 1 
| BARRIER I 
!M BARRIER I 


CREATE 
AWARENESS 
& 

PROMOTE 
PROGRAM 

7 


I BARRIER J 





r BARRIER 
REDUCTIONS 



BARRIERS 

Lack of funding at key points 
in time. 

Unrealistic time lines. 



Staff/volunteer overcommitment. 



Lack of access to media. 



Lack of promotional expertise. 

Failure to recognize need for 
promotional efforts. 



BARRIER REDUCTIONS 

Seek other sources/ 
resources; reduce efforts. 
Adjust time lines, 
overtime, improve 
planning, reduce offerings. 
Delegate, provide 
overtime, reassign 
responsibilities. 
Use alternative marketing 
strategies (e.g., direct 
mail); develop relationship 
with media. 
Provide training, gain 
access to expertise. 
Provide training. 
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Table 9.24. 



Barriers and Reductions for Main Events 6 and 8 



ACQUIRE 
i/OR 
TRAIN 
STAFF 
& 

VOLUNTEERS 



T BARRIER T 
I BARRIER 1 
1 BARRIER I 
=tf BARRIER I 
L BARRIER J 



BARRIER 
REDUCTIONS 



INITIATE 
NETWORKS/ 
COLLABORAT- 
IONS 



8 



J 



BARRIERS 



BARRIER REDUCTIONS 



Lack of funding at key points 
in time. 

Unrealistic time lines. 



Staff/volunteer overcommitment. 



Lack of access to media. 



Lack of promotional expertise. 

Failure to recognize need for 
promotional efforts. 



Seek other sources/ 
resources; reduce efforts. 
Adjust time lines, 
overtime, improve 
planning, reduce offerings. 
Delegate, provide 
overtime, reassign 
responsibilities. 
Use alternative marketing 
strategies (e.g., direct 
mail); develop relationship 
with media. 
Provide training, gain 
access to expertise. 
Provide training. 



9 
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Table 9.25. Barriers and Reductions for Main Events 6 and 9 



ACQUIRE 
&/0R 
TRAIN 
STAFF 
& 

VOLUNTEERS 



T BARRIER 1 



1 BARRIER I 
| BARRIER I 
=N BARRIER I 
L BARRIER J 



PROVIDE 
EDUCATIONAL 
EXPERIENCES 



BARRIER 
REDUCTIONS 



BARRIERS 

Unrealistic timelines* 

Inadequate linkages to 

other agencies* 

Lack of local staff "buy-in." 

Unresponsiveness of state 
specialists. 

Discrepancy between local need 

and program resources. 

Lack of local community "buy-in." 

Time lag between development 
and implementation.* 

Diversion of staff resources 
due to crisis** 

Insufficient enrollment/attendance* 
Bad weather* 



BARRIER REDUCTIONS 

Provide for overtime, readjust 
expectations. 

Increase communication with other 
agencies. 

Provide in-service training, 
administrative endorsement. 
Improve staff communications. 

Reallocate resources or find 
alternatives. 

Redefine need, improve marketing 
effort. 

Change program development 
process, design alternative 
delivery methods. 
Design alternative methods, 
seek outside resources. 
Improve marketing, problem/needs 
assessment and scheduling. 
Change delivery mode, reschedule. 
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Table 9.26. Barriers and Reductions for Main Events 7 and 9 



CREATE 
AWARENESS 
& 

PROMOTE 
PROGRAM 



F BARRIER 1 
| BARRIER I 
| BARRIER I 
=>| BARRIER I 
L BARRIER J 



PROVIDE 
EDUCATIONAL 
EXPERIENCES 



9 



f BARRIER ^ 
I REDUCTIONS I 



BARRIERS 

• Public apathy/alienation. 

• Vested interest in status quo. 

• Individuals'/communities' 
resistance to change. 



BARRIER REDUCTIONS 

• Diagnose causes and 
develop remedies. 

• Educate individuals/ 
communities regarding 
topical area. 

• Educate individuals/ 
communities regarding 
topical area. 



See also the single starred items in 6 to 9 in Table 25. 
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Table 9.27. 



Barriers and Reductions for Main Events 8 and 9 



INITIATE 
NETWORKS/ 
COLLABORAT- 
IONS 



f BARRIER 1 



f BARRIER i 
| BARRIER I 
z# BARRIER I 
L BARRIER J 



PROVIDE 
EDUCATIONAL 
EXPERIENCES 



BARRIER 
REDUCTIONS 



BARRIERS 



BARRIER REDUCTIONS 



See the single and double starred items in 6 to 9 in Table 25. 
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Table 9.28. Barriers and Reductions for Main Events 9 and 10 



INVOLYE 
RESEARCHERS 




i 


10 





,A R 
'? 

' R 
L. 



6B5B 
A A A A | 

R " 
R 
I 
E 
R 



V- 
V 

.J 



( 



=&/ BARRIER 
REDUCTIONS 



PROVIDE 
EDUCATIONAL 
EXPERIENCES 



BARRIERS 

• Problem/topic of little 
interest or low priority 
for researchers. 

• Conflict in orientation to 
time of Extension versus 
research. 

• Different criteria and standards 
of performance for Extension 
vs. research. 



BARRIER REDUCTIONS 

Increase level of interest/ 
priority through joint 
appointments and/or 
discretionary Extension 
funding. 

Plan for lead times; to 
involve researchers 
earlier; plan for use of 
preliminary/interim results. 
Seek joint appointments 
and composite standards 
for pay, promotion and 
tenure. 
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Table 9.29. 



Barriers and Reductions for Main Events 9 and 1 1 



PROVIDE 
EDUCATIONAL 
EXPERIENCES 



f BARRIER 1 



| BARRIER | 
I BARRIER I 
=tf BARRIER | 
L BARRIER J 



BARRIER 
REDUCTIONS 



CHANGES IN 
CLIENTELE: 
KNOWLEDGE 
ATTITUDES 

SKILLS 
ASPIRATIONS 
11 



BARRIERS 

• Limited attendance. 



Discrepancy between client 
expectations and program content. 

• Denial of the nature of the 
problem, feeling of not in 
control of destiny. 



Failure to accept responsibilities 
for problem/decisions. 

• Unrealistic expectations. 

• Self-service selection results in 
wrong program. 

Lack of active participation. 



BARRIER REDUCTIONS 

Use endorsement and incentives, give 
greater emphasis to personal 
benefits. 

Pre-survey to determine 
expectations; give more complete 
explanation of content. 
Portray problem as opportunity; 
use successful examples as 
role models; recognize past 
accomplishments; set achievable 
short-term goals; recognize time 
required to address problem; set 
priorities. 

Emphasize need to accept responsibility. 

Convey realistic expectations (time 
required, use successful examples). 
Learn more regarding participants 
beforehand; give more detail on nature 
of problem. 

Reorganize materials; use more 
exercises, alternate delivery modes. 



ERIC 
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Table 9.29. (continued) 



• Insufficient exposure. 

• Inappropriate mode of delivery. 
Poor instruction. 

• Poor instructional environment, 
curricular short comings. 



Use alternate delivery modes (e.g., tapes, 
news letters, group discussions). 
Evaluate and follow-up; use different 
mode(s). 

Improve screening, training, and materials 
development. 

Improve planning and control; evaluate 
and follow-up; improve understanding of 
learners; assess participants needs, 
train staff/volunteers to deal with hostile 
environment. 
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Table 9.30. Barriers and Reductions for Main Events 11 and 12 



CHANGES IN 
CLIENTELE: 
KNOWLEDGE 
ATTITUDES 
SKILLS 
ASPIRATIONS 
11 


r BARRIER 1 


CHANGES IN 
CLIENTELE: 
BEHAVIORS/ 
PRACTICES 

12 


I BARRIER I 
| BARRIER | 
A BARRIER | 
L BARRIER J 



r 

BARRIER 
REDUCTIONS 



J 



BARRIERS 

• Adherence to perceived norms, 
tradition. 

Resistance to change. 

Attitude that "it won't work 
with me/us--l/we are 
different." 

Feelings of apathy, lack of 
desire to change. 
Lack of motivation. 



Peer pressure against change. 



Incomplete learning process. 

Lack of individual empowerment. 
Different individual time frames 
for change. 

Preoccupation with day-to-day 
operations. 



BARRIER REDUCTIONS 

Change perceptions, provide 
support/networks for new ways 
of thinking. 

Change perceptions, provide 
support/networks for new ways 
of thinking. 

Use successful examples; 
establish networks; involve 
nay sayers; provide 
individualized attention. 
Use succesful examples and 
reinforce benefits of change. 
Use successful examples; 
emphasize smaller successes 
and personal benefits. 
Obtain endorsements; use 
successful examples; mobilize 
positive forces (e.g., create 
a group to support change). 
Evaluate and modify curriculum; 
follow-up. 

Create support network. 
Allow for it but follow-up with 
participants. 

Emphasize benefits from 
changing; restructure perceptual 
habits. 
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Table 9.30. (continued) 



Insufficient priority. 
Resource limitations. 



Difficulty in obtaining risk 
capital (if applicable). 
• Fear of risk-taking. 

Perceived risks exceed benefits. 

Lack of technical and personal 
support. 

Local, state and Federal 
regulations (as applicable). 
Lack of community loyalty 
(if applicable). 

Shrinking demand for products 
or services (if applicable). 



Use peer pressure (e.g., 
bandwagon effect). 
Help explore alternatives and 
set priorities. 

Develop legal support; form risk 
capital pools. 

Use successful examples; help 
evaluate risk/benefit realistically. 
Use successful examples; help 
evaluate risk/benefit realistically. 
Provide individualized attention 
and follow-up. 

Seek revisions or exceptions. 

Provide support for leadership 
development. 

Identify and focus on market 
niche. 
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Table 9.31. Intervening Events for Main Events 12 and 13 



CHANGES IN 
CLIENTELE: 
BEHAVIORS/ 
PRACTICES 



12 



' INTERVENING ' 
M EVENTS H 



I 



I 

J 



CLIENTELE 
COMMUNITY 
& 

SOCIETAL 
CONSEQUENCES 

13 



INTERVENING EVENTS 
(As Applicable) 



• Adverse weather conditions and natural disasters 

• Changes in interest rates 

• Boycotts 

• Changes in public policies (e.g., farm programs) or government regulations 

• Increased or new competition 

• Loss of major industry, business or institution 

• Changes in transportation routes or methods 

• Bank failure(s) 

• Loss of population 

• Environmental constraints 

• Loss of essential community services 

• Inventory loss to theft or damage 

• Changes in consumer behaviors 

• Life style changes 

• Fluctuations in local economy 

• Adverse environmental effects 

• Lack of community recognition of benefits 
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10. Developing the Program 



In this chapter we carry the illustrative program design from the preceding chapter into 
a developmental phase. Five modules and their tasks are identified in order to develop 
the educational experiences with their supporting materials, staff, recruitment and 
marketing efforts. A final task integrates the results of these efforts. 

10.1 Tasking the Developments Modules, Designating Timelines and 
Responsibilities 

There are five modules to be developed in this cycle or stage which follow directly from 
the illustrative program of the preceding chapter. The major module on which all others 
are dependent is the one which determines what the Educational Experience Profile will 
be. [ This involves main event 9 in Figure 9.1 and Table 9.9 of the illustrative program.] 
Only when it is clear what the structure and some of the content of this module will be, 
can developmental work begin on the other four modules of: acquire and train staff; 
create awareness and promote program; initiate networks and collaborations; and, identify 
and communicate the accomplishments of the program. Of these latter, work should 
probably begin earliest on the network and collaborations module and last, on the 
accomplishment module. The dependencies among these modules are illustrated .in 
Figure 10.1 . Of course, for any actual program these dependencies may differ somewhat. 

We may recall from the previous chapter that some participants from the design stage 
also participate as members of the development team. In effect, they serve as linkage 
experts to work done earlier. Added to this core are others who bring specialized skills 
and expertise to the developmental stage. Some of them may be from other organizations 
or agencies. The question is how relationships among members of this group should be 
structured so as to optimize their functioning. We suggest that one person have the 
responsibility of coordinating the group's efforts and reporting to the administration 
periodically on their progress. In addition, that this person be vested with some authority 
over the work that the group performs but that a major point not be made of this since 
the desire is to maintain harmonious and collegial relationships. It is also recommended 
that a facilitator be used, especially early on to help articulate the tasks, timelines and 
responsibilities. Again, it is desirable that this person be a third party to the process and 
have some evaluation/research training. S/he may be the same facilitator as from the 
design phase. In effect, the group may be thought of as engaging in a mini-design for the 
developmental cycle. The group may also be thought of as the Life Cycle Guidance 
Team for this phase of the program. A prototype agenda for their initial meetings might 
appear as in Table 10.1. 

In the first activity on the agenda it is anticipated that those members who participated 
in the design phase would play a key role in helping to explain to the newcomers the 
various concepts and nature of what was accomplished in the design phase. Of course, 
if the facilitator participated in the design phase then s/he can also play an important role 
in this orientation process. First, the target audiences are grouped into fairly 
homogeneous sub-groups based upon the similarity of their KASAB's. Although there is 
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Table 10-1 Prototype Agenda for the 
Development of Program XYZ 
in a One to Two Day Work Session 



XTIME 
(APPROXIMATE) 



20 



# 

1 



ACTIVITY 



REVIEW DESIGN RESULTS 



45 



DETERMINE THE EDUCATIONAL EXPERIENCE 
PROFILE(EEP): A FACET APPROACH 

- GROUP TARGET AUDIENCES' KASAB'S 

- REVIEW OF A FACET APPROACH TO 
PROGRAM DEVELOPMENT 

- SPECIFYING THE EEP IN TERMS OF 
THE FACETS 

- DETERMINE CLIENT PROBLEM SOLVING 
STEPS &/0R FOCUS GROUPS TO 
FORMULATE &/0R CHOOSE AMONG 
ALTERNATIVES 

- DESIGNATE TASKS & TIMELINES FOR 
THE EEP MODULE 



30 



DESIGNATE TASKS & TIMELINES FOR THE 
OTHER MODULES OFt 

- DEVELOP RECRUITMENT PROCEDURES 
& TRAINING MATERIALS 

- DEVELOP PROMOTIONAL MATERIALS 
& PROCEDURES 

- ESTABLISH NETWORKS 
& COLLABORATIONS 

- DEVELOP ACCOMPLISHMENT REPORTING 
& DISSEMINATION PROCEDURES 



MAKE PLANS FOR SUBSEQUENT MEETINGS 
TO REVIEW & REPORT ON PROGRESS 
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o Content 



The development team may obtain guidance of a general nature from the Matrix of 
Educational Effects concerning what the content of the program should be. Further 
specification of thesr iffects may be necessary in order to guide the development of new 
materials or the selection and/or adaptation from those already available. Such materials 
can be thought of as being developed or adapted in the following three ways: (1) a 
standard or uniform approach in which the materials are to be used in the same manner 
whenever or wherever the program is carried out; (2) a modular approach in which 
different parts or modules can be used in different combinations as appropriate to the 
circumstances; (3) a modular approach with explicit provisions for tailoring them to 
meet the needs of a particular target audience or locale. Tailoring can occur by 
incorporating indigenous persons or examples into the modules or by augmenting the 
modules with local concerns (e.g., interpretations by discussion groups or prominent 
persons as to how this affects us and why it is important for us). Both kinds of tailoring 
are possible for any given module. 

o Time 

Time or the temporal aspects of program development (Kelly and McGrath, 1988) focus 
on the presumed nature of the learning or proficiency acquisition curve and deal with such 
questions as: (1 ) what should the learning experiences be; (2) how frequently should 
they be provided; (3) how long should they last singly and/or in combination; (4) how are 
they best put in sequence; (5) how much time elapses between the learning experiences 
and the anticipated behavior change or their consequences; and, (6) can reinforcing 
experiences be provided to support and sustain the acquisitions that have been made for 
this intervening period of time, if it is very long? 

Some of the preceding questions imply that a progression model could be developed of 
the stages that typical members of the target audience go through in acquiring the 
KASAB's for their problem-solving needs (Boone, 1985; Smith, 1989a; Van den Ban and 
Hawkins, 1988). This idea is pursued further in Chapter 1 4. 

o Context 

The context in which or through which a learning experience takes place can vary widely 
and may be thought of as involving three main categories: (1 ) the mechanisms involved 
to deliver the program; (2) the means by which or through which they are used; and, (3) 
the settings in which they take place. Examples of each category are given in Table 
1 0.2. These examples are meant to be illustrative rather than exhaustive. They do serve 
to illustrate the large number of combinations that are possible. Reiser & Gagne* (1983) 
speculate that the lack of evidence concerning the differential effectiveness of different 
delivery mechanisms may be due to their not being chosen on the basis of how well they 
would contribute to a given set of learning objectives. They show how the extremely large 
number of combinations, such as those given in Table 10.2, can be reduced dramatically 
by a sequential selection process and they provide an algorithm for doing so. 



some subjectivity in these judgments they have to be made in order to reduce the sheer 
number of groups involved and Introduce some economies into the developmental phase. 
Of course, if there are very few target groups and they are distinctly different then such 
a grouping may not be necessary nor even desirable. Next, a facet conceptualization of 
program development is reviewed. Then, the group starts detailing tasks for the 
development of the Educational Experience Profile (EEP) and their implications for the 
tasks of the other modules. Finally, dates for next meetings to review and report on 
accomplishments are set. 



10.2 Determining The Educational Experience Profiie: A Facet Approach 

For non-formal educational programs such as those dealt with by Cooperative Extension, 
the developmental phase of a program will usually focus on what are often called the 
delivery mechanisms and curricular materials to be used in reaching different target 
audiences. For any program development phase there are at least four facets which play 
back and forth on one another as the developmental phase unfolds. They are: (1) the 
resources available to carry out the program; (2) the content of the curriculum or 
curricular material; (3) the temporal aspects such as in what order the sequence of 
learning experiences will be provided, for how long and how often and, how much time 
will elapse between these experiences and the hoped for behavior changes or their 
consequences; (4) the context in which or through which these learning experiences will 
take place. The latter are sometimes referred to as "methods" although they really 
embrace a wide range of alternatives from the physical aspects of a learning experience 
through the people aspects to the organizational aspects-we shall have much more to 
say about these later. These four interdependent facets carried to completion for a given 
target audience's (or group of audience's) KASAB's are called the educational 
experience profile (EEP). Some aspects of the profile will have been indicated in the 
design phase; however, their detailed specification doesn't really take form until the 
developmental phase. 

o Resources 

r 

Decisions made in the developmental phase can dramatically affect the costs of the 
implementation phase. Hence, it is not unusual to hear development team members 
articulate the interplay by saying "going that route would be too expensive later on so we 
had better find another, less costly alternative." The referent can be anything from 
promotional materials or equipment needed to staff considerations. Of course, the team 
can always specify different levels of effort for the implementation phase or an 
incremental phasing in of the program. Alternatively, they can even postpone aspects of 
the developmental phase until certain events or components of the program are phased 
in. However, the latter usually runs the risk of other circumstances overtaking these 
efforts and the program never seeing the "light of day." For our purposes we shall 
assume that all of the developmental work will be completed before the implementation 
phase is begun. 
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In Table 10.2 Folk Media involve the use of itinerant actors, singers or storytellers (Van 
den Ban and Hawkins, 1988) while Campaigns were used in the early years of Extension 
work in particular (Rasmussen, 1989; Mayberry.1990) wherein local staff lay the 
groundwork for a visit by a campaign team, the campaign is conducted and the 
momentum is initiated by the team, with follow-up provided by local staff. Extensive 
information on volunteer programs and how to form them can be found in Steele et. al 
(1989). If time permits and the team is interested they can enumerate their own 
specialized lists such as done by Jones (1992) for critical thinking. 



o Research/Knowledge Base 

A facet of orogram development that is not generally recognized or utilized is the explicit 
use of a research/knowledge base to guide the myriad inter-related decisions that must 
be made. Extension in particular, with its home in the Land Grant University System, 
prides itself on the development of research-based or linked programs. Indeed, explicit 
reference to the research and knowledge base should guide the other four facets as they 
interplay in the developmental process. Pictorially these five facets and their 
interrelationships can be depicted as a cone in which the research and knowledge base 
(R/K) is at the apex of the cone with the other four facets forming points on the circular 
base Dynamically then, 'the developmental process can be thought of as a spiral or 
sequential model (Van den Ban and Hawkins, 1988) in which the cone gets progressively 
smaller as different developmental decisions are made, as follows: 



where: R = Resources; T = Time; C = Content; C = Context; and, 
R/K = Research/Knowledge Base. 

Examples of the kinds of things that would form part of the research/knowledge base are: 
evaluations of related programmatic efforts; topically relevant publications reporting the 
results of research; results of demonstrations or field trials some of which may have been 
especially designed to deal with the topic; expert opinion based on extensive research 
and practical experience; etc. 
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A cone mav encompass several cells or even an entire column The deciding facto is 
wh^teveMhe ^developmental team finds is easiest to facilitate their thinking abou such 
maS Thfs wTusSJly entail the columns of the matrix. Once the profile(s) has (have) 
SKrtnad the resultant matrix may look like the following wherein a single column 
contains the results of the decisions made for the four facets (R; O, T, O). 

TARGET AUDIENCE 

LIMITED RESOURCE 
HOUSEHOLD MEMBER 



RESOURCES: 
CONTENT: 



TIME: 



CONTEXT: 



Funds for 10 paraprofessional salaries plus 
travel plus printing of modules. 

Modular with modules for: sanitation and 
personal hygiene; food purchasing; food 
preparation; food storage; clothing repair; 
money management; parenting. 

One visit per week for 2 hours each for 36 
weeks (1 on 1 ). 

Pictorial materials in color to be used by the 
paraprofessional with the homemaker in his/her 
private dwelling. 



This matrix will in turn lead to a series of SP^ 6 ^ 

team members or their designates. As an aid in specifying the EEP or in the course ot 
Sna o^t these tasks subsequent to its specification-, a variety of techniques may be 
KlwS to those involved in program evaluation ™*™*?™** Q ™ 
have been used by them for many years. Examples of some of these efforts and 
methods that might be used are given below: 



Purposes 



Methods 



To determine which means are most effective for reaching Surveys &/or 

or recruiting members of the target audience(s). S^n%89) 



To determine the relevance and appropriateness of 
the content to members of the target audience(s). 



Individual &/or 
group interviews 



io-i<i84 



o The Profile 



Once the target audiences have been grouped by the nature of their KASAB's, 
developmental efforts can be oriented toward these larger groupings still utilizing the 
matrix approach. In the illustration that follows each section of the matrix can be thought 
of as having its own cone: 

TARGET AUDIENCES 



KNOWLEDGE 
ATTITUDES 
SKILLS 
ASPIRATIONS 



BEHAVIORS/ 
PRACTICES 



A.C&D 



B.F&E 



K.S&Y 



Chapter 14 gives examples of stage - process models and a "theory of learning" that the 
development team may want to use as a guide in sequencing the EEP. 
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s 



To determine the appropriateness of the structure 
and format of the content to the target audience(s). 



Individual &/or 
group interviews 



o 



To ascertain the degree of importance to attach to or 
amount of time to allocate to different content areas 



Expert opinion.or 
Group Process 
techniques 
(Moore, 1987) 



o 



To determine the accuracy of the curriculum content. 



Peer review 



For example, the use of individual interviews to obtain information from clientele so as to 
determine the content of the program was cited earlier in Guam's work on Import 
Substitution (Workman, 1988). Similar uses might be made of surveys. Krueger (1989) 
cites the use of focus group interviews to help determine the structuring and educational 
format for a nutrition education program. Wade (1993) used focus groups to aid in 
determining the delivery mechanisms and program content for a program on Adolescent 
Pregnancy & Parenting. 

10.3 Specifying the Developmental Tasks for the Educational Experience Profile 

Once the EEP has been specified the following tasks can be conducted: 
o Search For & Review Relevant Materials 

In order to avoid later duplication a single, all encompassing search is made of materials 
relevant to all of the developmental modules, not just those for the EEP. However, 
detailed review of the materials relevant to the other modules may be held in abeyance 
until the EEP becomes better known. After review of -the EEP related materials decisions 
are made as to which ones can be used or adapted and for what topics developmental 
efforts are needed. One step that can aid these decisions as well as those of the next 
task is to sort the KASAB's from the Matrix of Educational Effects into different content 
areas, as depicted in Figure 10.3, since such materials are usually archived in that 
manner, anyway. The specification of learning objectives for these content areas a la 
Gagne' et. al (1992) may also help guide the next task. 

o Adapt &/or Develop Relevant Materials for the EEP 

In this task the relevant materials will be adapted or for those topics/objectives for which 
materials are lacking - some will have to be developed. Considerable time and resources 
may be required for this task if much developmental effort is involved. Performance 
contracting with time as one of the criteria may be one way of accelerating such efforts. 
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o Review &/or Pre-test Resultant Materials 

The materials that result from the pror task may require review by subject matter experts 
and some kind of a pre-test, especially if the materials are very new. Expert opinion and 
focus groups with potential clientele are two such means for achieving these. 

o Finalize & Produce the EEP Materials 

In this task the materials are revised based upon the findings of the preceding task, put 
in a form suited for the delivery mechanisms and produced in sufficient quantity for the 
implementation phase.[ Alternatively, the reproduction in large numbers may be held in 
abeyance until the completion of the other modules, especially if it is anticipated that their 
development might result in some modifications to the EEP materials.] An illustrative set 
of timelines for these tasks is given in Figure 10.3. 

The developmental efforts for the other related modules can begin at times that are keyed 
to the time-phased accomplishments of the EEP module. Conceivably, some of these 
modules would go through the same sequence of developmental steps subsequent to the 
overall search. However, the collaboration and accomplishment modules would not likely 
follow this same sequence of steps, as we shall see. 

10.4 Develop Materials & Procedures for Acquiring & Training Staff & Volunteers 

In this module materials will be developed to acquire and train those who will provide the 
program. A critical question that will affect the extent of training relates to how much prior 
experience with the organization, the subject matter, other program providers and 
clientele of the proposed type, do staff &/or volunteers bring to the setting. Those with a 
great deal of experience may need little more than a brief o^entation while for those with 
little or no experience training alone may not suffice - rather some kind of mentoring of 
the more experienced with the less experienced may have to be put in place as part of 
the ongoing program (Zimmer & Smith, 1992). The Program Logic Model and It's 
components should serve as an Important vehicle for orientation to the specifics 
of the program including intended effects. Tasks for this module might appear as 
follows: 

o Prepare Job Description & Announcement of Need for Program Providers 
o Develop Brief Screening & Interview Procedures 
o Prepare Invitation &/or Contracts 
o Prepare Training Agenda - including 

o Nature & Intended Effects of Overall Program 

o Clientele Recruitment Procedures 

o Nature of Delivery Teams & Roles of Members 

o Uses of Instructional Materials 

o Collaborations & Referrals 

o Promotional Efforts & Materials Use 

o Accomplishment Reporting 
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o Procedures & Materials for Trainee Feedback 
o Prepare & Produce Materials for Training Agenda 
o Identify Trainers & Prepare Schedule(s) for Training Sessions 

The use of participant observers (perhaps to whom other participants give their 
comments) coupled with structured group interview techniques is a powerful 
alternative to the usual end-of-training form and often yields more useful 
information. 

An illustrative set of timelines for these tasks is given in Figure 10.4. 

10.5 Develop Materials & Procedures for Creating Awareness & Promoting the 
Program 

In this module promotional materials for the start-up of the program will be prepared that 
can be used as it is initiated in different sites. There is also a provision for periodic 
feedback on program accomplishments to targeted audiences, administration and select 
stakeholders. The results obtained from the accomplishment reports will form part of the 
basis for this feedback, including local success stories. The "personal touch" with select 
stakeholders through staff, volunteers and clientele is thought to be particularly effective 
and will be relied on heavily in the conduct of Main Event #7:" Create Awareness & 
Promote Program". Some aspects of this approach may need to be included in the 
training sessions for staff and volunteers. Steps in this module might run as follows: 

o Review Materials from the Search 

o Adapt &/or Develop New Materials & Procedures (as appropriate) for: 

o Brochures for Handouts & Mailouts 

o Briefings 

o Newspaper Articles 

o Public Service Announcements for Radio/TV 

o Promotional Events at malls, fairs, ete: 

o Forms of Recognition for Support &/or Involvement 
o Develop Format, Proposed Content & Utilization Procedures for the Local 
Newsletter 

o Develop Procedures & Materials for Feedback from Newsletter 
Recipients 

An illustrative set of timelines for these tasks is given in Figure 10.6. 

10.6 Establishing Networks/Collaborations 

Many of the activities listed for Main Event #8 "Initiate Networks/Collaborations" can 
actually begin in the developmental phase. Also, they probably need to since 
arrangements with different bureaucratic agencies can be quite time consuming and may 
require some protracted negotiations, especially where staff and resources are concerned. 
If other agencies are to participate in specifying the details of the program then they 
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would likely have some representation in the design and developmental phases. Such 
involvement might actually facilitate later collaborations. However, this Main Event refers 
only to laying the groundwork for a collaborative effort once implementation gets 
underway. For the latter, some specifics need to be known about the program so that the 
. nature of other organization's involvement can be discussed and negotiated. It would 
seem prudent to establish contact with the other organization(s) as early in the 
developmental phase as possible and complete as many of the following activities as is 
practical: 

o Establish Contact & Solicit Cooperation 
o Obtain Commitment 

o Negotiate Roles & Responsibilities including Staff & Resources 
o Identify the Key Actors &/or Products Involved 

o Identify the Nature & Schedule for their Involvement (recognizing that some 
of their staff may participate in the training sessions) 

An illustrative set of timelines for these tasks is given in Figure 10.7. 

10.7 Develop Accomplishment Reporting Procedures, Materials & Dissemination 
Efforts 

To meaningfully devise a reporting system for program accomplishments one must know 
not only the structure and content of the program but the implementation and expected 
participation rates for different groups of clientele. In the implementation cycle the focus 
will be primarily on getting the program "up & running". Whereas in the 
maintenance/improvement cycle the focus will shift to clientele benefits and the 
identification and infusion of exemplary practices into the program. However, as we shall 
see, the implementation cycle may provide an opportunity for obtaining some "early 
returns" on clientele benefits, if the rate of implementation is of a certain type. These 
notions are discussed in the tasks that follow: 

o Identify Implementation Schedules 

Many programs have a graduated schedule for implementation that looks very much like 
a "learning curve", as in Figure 10.8, wherein a few sites are the initial implementors with 
others following. Such a curve can provide an opportunity to learn about the problems 
encountered in implementation and their resolution early on. Since these early sites reach 
a state of maintenance sooner than the others they can also provide an early view of how 
participants are benefitting. Other schedules also depicted in Figure 10.7 illustrate the "all- 
at-once", "staircase" and "ogive" schedules. The latter two also provide an opportunity for 
learning and "early returns" as well as a respite or delay period which can additionally be 
used for problem identification and resolution. Since all of these are variants of the 
learning curve, we shall focus primarily on it. 

For the graduated implementation schedule two mechanisms are suggested to enhance 
problem identification and resolution. They are: 
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submitted in a form that allows ready compilation and aggregation thereby enabling the 
depiction of total program implementation quickly. Feedback, in profile form, will be 
provided to each site depicting their status and accomplishments relative to the total 
program. This form, when augmented by other local information will provide the basis for 
the newsletter or progress report which will comprise one of the dissemination efforts. 

o Designate Maintenance & Improvement Variables 

Once a "steady state" has been attained the number of sites and/or the number of 
participants becomes a constant while the information on the other variables reflects the 
periodic performance of the program, with corrective actions being taken as deemed 
necessary by staff. Focus now turns to clientele benefits and the identification and 
diffusion of desirable practices. Clientele benefits are best obtained by a 3rd party follow- 
up (perhaps by phone) with participants, former participants, dropouts, etc. to determine 
their perception of benefits, problems and improvements that might be made, etc. This 
3rd party should not have any vested interest in how the program has been carried out 
nor any felt need to defend it against what might be unfair or uninformed criticisms nor 
any motivation to put the results in a favorable light. If these conditions can be satisfied 
then this 3rd party may be a volunteer, a student, a staff member of some other program 
or site, etc. This foliow-up would focus on perceived benefits or surrogates for benefits 
that are longer term in nature, might employ a sampling rationale( if the # of participants 
to be contacted is large), and would provide aggregate information back to the site devoid 
of any identifiers of individual respondents. Such information would form one source for 
the identification of desirable practices. Another source would come from local staff on 
what they deem their most successful efforts to be in terms of effectiveness, efficiency 
or other forms of exemplariness with these submissions being reviewed, codified, 
synthesized and promoted centrally. Incentives might be provided to encourage adoption 
of particular practices with such adoption being reported in the periodic reports of the 
reporting system. 

A note about the identification of successful practices may-be-appropriate. One approach 
is to have staff identify from their personal observations, practices that they consider 
unusually effective, efficient or desirable for some other reason and, attempt to 
systematize them and infuse them into the program. This also might be attempted for 
ineffective practices in order to identify training needs or structural impediments in the 
program. However, staff are often reluctant to identify "undesirable" practices either about 
themselves or others. Structured group interviews might serve as a useful tool for 
identifying such practices, especially for the latter. [See Chapter 1 4 for one such approach 
using performance incidents.] Another approach is to "borrow" what are deemed desirable 
practices from other sites, programs, organizations, etc. and try them out with one's own 
program on a limited scale to determine their effects. Staff might even invent new 
practices that they can then try out to test improvements. 

If, at some point in time after maintenance is attained, it is deemed necessary or 
desirable to do an evaluation of the program, then procedures like those outlined in 
Chapter 15 and/or 16 could be followed. 
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oo Mentor Sites - early implementors can provide guidance to later 
implementors thereby facilitating the ease of implementation 

oo Hot Line - a "hot" line can be established to provide guidance and 
consistency to the manner in which problems are resolved. Such might 
be particularly useful for the "all-at-once" implementation schedule 

We need also to recognize that there is an expected time to complete implementation and 
arrive at the maintenance state and that this expected time can vary considerably from 
one site to another, depending on a variety of contextual variables. Examples of 
contextual variables range from such obvious ones as geographic distance, staff 
shortages, intransigence of a particular group of clientele or the severity of other problems 
impinging on them, etc. Pursuing the axiom that "nothing succeeds like success" it might 
be prudent to start with the sites where success comes quickest and then move on to the 
others. However scheduled, experiences from the early sites may help reduce the 
expected time for later sites. 

o Designate Implementation Variables 

For many of the implementation variables there is an expectation or expected value and 
an observation or observed value. The magnitude of this difference between expected 
and observed forms the basis for judging how well implementation is proceeding. For 
example: 

oo Number of Sites (or Units) - the implementation schedule provides the 

expectation for the number of sites in which the program will be 

implemented, by time 
oo Attributes of Sites - local staff will have indicated the attributes of sites that 

they expect to be working in, within a specified time period 
. oo Number of Participants Per Site - local staff will have indicated the number 

of participants they expect to be working with, within a specified tim>? period 
oo Attributes of Participants - local staff will have indicated the attributes of 

participants they expect to be working with, within a specified time period 
oo Nature & Extent of Participation by Attributes of Participants - local staff 

will have indicated their expectation for the extent to which different types 

of clientele will participate in different aspects of the program, within a 

specified time period 

Variables such as these will form part of the reporting system by which local staff will 
forward to a central location information about the site's (or unit's) conduct of the 
program. Other information included in the reporting system will pertain to such 
implementation concerns as are spelled out in the activities and indicators for the Main 
Events in the Program Logic Model. Additional information will pertain to the availability 
and accessibility of resources necessary to carry out the program, work with other 
agencies, problems encountered and resolved, etc. Site or unit reports will be submitted 
on a periodic basis (e.g. weekly, monthly, quarterly) with problems on accuracy of 
reporting handled through supervisory oversight. Implementation information will be 
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o Identify Reporting & Dissemination Procedures 

The primary means of dissemination will entail a follow-up either in person or by mail or 
phone, with and through the means utilized in the module on "Create Awareness & 
Promote Program". The mechanisms would be the local newsletter with personal success 
stories and the feedback report cited earlier. 

An illustrative set of timelines for these tasks is given in Figure 10.9. 

10.8 Integrating & Finalizing the Products & Processes 

The final task of the developmental phase is not meant to be very involved or time 
consuming. The ease or difficulty with which it is completed will depend very much on 
how well the developmental team members have coordinated their earlier efforts for the 
different modules. [This effort is listed as a task rather than a module because it involves 
the summation and integration of the modular results and should occur over a fairly short 
time period (e.g a few days to a few weeks).] 

Once the developmental modules have been completed and an actual profile exists for 
the target audiences their "readiness" can be reviewed or tested prior to full scale 
implementation by use of some of the following: 

- Structured Group Review using subject matter and "lay" experts 

- Role-playing 

- Field tr.ils 

For example, in her work on Water Quality in W and, Smith (1989b) used field trials 
to test materials and approaches in order to develop an identifiable program. 

Alternatively, coordination among the different module development efforts may have 
been so thorough that their integration is a relatively minor effort and, if further testing is 
not required they can proceed to the implementation phase. Figure 10.10 attempts to 
portray these integrative efforts. 

In view of the central role that the Program Logic Model and it's components will 
play as an orientation and training tool, it may have to be updated or revised due 
to decisions made and actions taken in the developmental phase. 

In the next chapters we see how these concepts and procedures are put into practice. 
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Figure 10.10 Integrating & Finalizing 
the Products & Processes 
from the Developmental Modules 
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11. Initiating the Program and Monitoring implementation 

in this chapter we focus on such concerns as the assessment of readiness for 
- implementation, publicity efforts that may be associated with the "kickoff" of the program, 
indicators of implementation, problems encountered and resolved, criteria for the 
attainment of implementation and early indicators of effects on clientele. But first we 
examine the role of the Life Cycle Guidance Team (LCGT) in such an effort. 

11.1 What is the Role of the Life Cycle Guidance Team in Implementation? 

In this stage of the program the Life Cycle Guidance Team (LCGT) will be concerned 
primarily with the rate and adequacy of implementation. However, they will also examine 
the program's readiness for implementation, attend to any publicity efforts that may be 
planned as part of the initial efforts and see to other matters that are necessary for the 
sustenance of p/ogram plausibility. 

11.2 Checklist Analysis of Readiness for Implementation 

Before proceeding with implementation the LCGT may want to reassure itself that its* 
newly developed program is worthy of implementation. A number of criteria have been 
suggested for judging the quality or excellence of programs. Two authors in particular, 
have put forth criteria forjudging Extension programs but these criteria undoubtedly apply 
to other organizations as well. Both sets of criteria encompass the notion of the "life- 
cycle" of a program. Those of Mueller (1991) are more explicit while those of Smith 
(1992, 1991) tend to be embedded in the concepts that encompass the criteria. These 
criteria have been adapted somewhat to better apply to the stage just prior to 
implementation. They can actually be used at any stage of the life-cycle process to gauge 
adequacy of accomplishments to that point. 

Table 11.1 gives an adaptation of Mueller's quality indicators.-They areorganized oy 
program phases (problem selection, commitment, strategy implementation, review/sunset) 
and by the central question being addressed in each phase (is it an important problem - 
in which we can make a difference - and recognize that we did). The quality indicators 
are examined to obtain an answer to each central question while the criteria pertain to 
different aspects of the indicators. 

Table 11.2 gives an adaptation of Smith's criteria for excellence. Her criteria are 
organized around the three major concepts of: relevance; quality; and, utility. Relevance 
is concerned mostly with the problem identification process while quality is concerned 
more with the nature of the program. Utility focuses more on the program effects and its 
consequences. 

Should the LCGT decide to use these criteria to judge their "about to be implemented 
program" and should that program resemble the illustration we have been using, it might 
find something like the following. On Smith's criteria for relevance and quality the 
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proposed program would fare well save for the efficient use of resources and evidence 
of goal achievement. It is presumed that the new program will be an efficient use of 
resources and some considerations of efficiency were dealt with in the developmental 
phase. However, a formal cost-effectiveness kind of analysis was not part of the process. 
Similarly, evidence of goal achievement doesn't become available until the program is 
underway.fActually, Smith (1992) indicates stages in the program for which different 
criteria are most appropriate.] 

The program to be implemented can also be judyed by the criteria from Mueller's first 3 
phases in Table 11.1. Again the new program fares well on these criteria save for those 
that pertain to timing, ethics and risks. Appropriate timing is presumed however, 
determining time limits for the program was not a necessary part of the developmental 
phase and may entail other concerns beyond the purview of the LCGT (e.g. the 
intransigence of the problem, it's incidence and ecology, future resources available, etc.). 
The ethical implications of addressing the problem and the risks/benefits of initiating a 
program to deal with it usually occur very early in the life-cycle process, somewhere 
between problem identification and the design phase. However, formal means of 
addressing these have not been incorporated into this process. Mueller's other criteria 
focus on program accomplishments and consequences. 

The LCGT may also bring other criteria to bear on their judgments of quality or excellence 
of the program at that point in time. Examples of these are: the extent to which the 
occurrence of potential barriers have been anticipated and their reductions planned for; 
the extent to which unplanned occurrences (e.g spin-offs) or intervening events would 
exacerbate the problem or work counter to the desired effects or consequences; the 
extent to which clients will have the necessary prior knowledge and skill to benefit from 
the program, etc. 

Assuming that the LCGT feels comfortable with the program-to-be, next steps in the 
implementation process can be dealt with. 

11.3 Inaugural Activities 

Activities may be planned to publicize and commemorate the initiation of the program. 
These are not the same as those planned for in the Create Awareness module cited 
earlier. Rather, these would be efforts that are intended to recognize the organization's 
overall involvement in the topical area with this new program as well as others it may be 
sponsoring. These efforts could range from nothing more than a news release to a gala 
kind of celebration with the involvement of a number of dignitaries. Alternatively, no such 
activities may be needed or desired. 

11 .4 To What Extent & How Well Are the Events & Activities Being Accomplished? 

In order to monitor implementation, the LCGT will depend in large measure on information 
that is provided by the program's reporting system, as developed in the module on 
accomplishment reporting. This would be augmented by information from the "hot" line 
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and mentor sites as well as site reports made by field staff supervisors.These latter would 
focus primarily on unit performance rather than individual performance. The general 
questions that the LCGT would concern itself with are: 

o What do the various sources of "feedback" tell us concerning implementation? 

o What barriers/problems have been encountered and how have they been 
resolved? 



o What are the implications of these answers for the expected rate and nature of 
implementation? 

Since the Program Logic Model (perhaps revised as a result of the developmental phase) 
serves as the "road map" for implementation, the above questions can be focused on the 
main events in the model that are concerned with implementation <uid with which 
barriers/problems can be associated (viz. Main Events 6 through 10 in the illustrative 
model). By organizing the results in this manner, the LCGT can make Diagnostic 
Decisions and recommend Corrective Actions concerning implementation. Examples of 
these are: 



Barrier(s) 

Some systematic 
segment of the 
target audience 
is not being 
reached(e.g. 
most needy, 
less affluent, 
minority. non- 
English speaking, 
etc.). 



Diagnostic Declsion(s) 



Examine policies & definitions 
to see if they are exclusionary. 
Ascertain if there are disincen- 
tives to participation (e.g what 
appeals to some may not to 
others or may even be repugnant). 



Corrective Actlon(s) 

Change policies/definitions 



Change approach or incentive 
structure. 



11.5 To What Extent is the Progranras Implemented the One Intended? 

Once implementation has been underway for some time the question arises as to whether 
or not the program as implemented is the one intended. Corwin (1983) identifies two 
kinds of changes that can take place in the course of program implementation: 

o Reconciliations - occur when those implementing the program cannot comply 
with its requirements but have to make some adjustments 

o Mutations - occur as a result of reinterpreting concepts in novel or creative ways 
due in part to the fact that they were expressed in abstract language instead of 
operational terms 
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Figure 11.1 Likelihood of Implementing an 
Intended Program by Degree of Specificity 
& Fidelity of Implementation 
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Scheirer (1987) reminds us that the faithfulness or fidelity with which a program can be 
implemented will depend in part on the degree of specificity it is given prior to 
implementation. In Figure 1 1 .2 we attempt to integrate these concepts into a common 
framework by portraying the likelihood that a program will be implemented as intended 
depending on its degree of specificity and fidelity of implementation. Examination of this 
figure shows that: 

* In quadrant 1 are found programs that are highly specific but can be adapted in 
any way desired in the course of implementation. If a specific program is the one 
desired then it seems highly unlikely that ft will be the one implemented. 

* In quadrant 4 are found programs that lack specificity but are to be implemented 
with a high degree of fidelity or exactness. This is really a contradiction in terms 
since ft is impossible to exactly reproduce something that lacks specificity. Hence, 
ft is also a highly unlikely result. 

* In quadrant 2 are found programs that have a high degree of specificity and 
hence could be reproduced exactly if such exactness were possible. But, C srwin 
(1983) suggests that such exactness may not be obtainable without allowance 
for reconciliations. Hence, a high likelihood of implementation as intended is 
qualified by the occurrence of a number of reconciliations. 

* In quadrant 3 one finds programs that lack specificity and are completely 
adaptable. Again we find an apparent contradiction in terms for a program lacking 
in specificity could only be implemented by each site inventing its own. In the 
extreme, all of the sites would be mutants since that is the only means by which 
they could gain the specificity to be implemented. Hence, the high likelihood of 
implementation is qualified by the occurrence of mutations. 

These considerations suggest that there is an optimal point between specificity and 
fidelity that will not be so specific as to leave implementing sites with no possibility of 
adaptation nor so general as to put them all at risk of "doing their own thing". In such a 
case some reconciliations will occur but they can be regarded as "informed adaptations" 
made with consultations wfth mentor sites &/or via the "hot" line. 

There are a number of factors that will affect just where this point of "optim&lity" is 
located. If the staff have a great deal of prior experience in carrying out similar kinds of 
programs then a high degree of specificity may not be necessary or even desirable. 
Indeed, to tie the hands of an experienced person may even be counterproductive. 
Alternatively, if there is no prior experience then some considerable degree of specificity 
may bt both necessary and desirable, perhaps coupled with mentoring and feedback 
from early implementors.lf there is an urgency for implementation then time may not 
permit for the development of specifics. An inexperienced staff coupled with a lack of 
specificity could lead to dire results unless some appropriate efforts are made to 
coordinate and calibrate their activities. An organization's tolerance for reconciliations 
versus mutations may affect the location of the optimum. This tolerance reflects to 
whom and for what they are accountable. If they are to foster innovations, then mutations 
are likely more tolerable than are reconciliations. Conversely, if they are to carry out a 
pre-defined model, then reconciliations are probably more tolerable than are mutations. 
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Finally, an organization's cultural values concerning conformity versus individualism 
may play an important role in determining just how much specificity is desirable or even 
tolerable. Organizations that have historically placed a premium on individualism may 
have great difficulty in obtaining adherence to a specific plan while those that value 
conformity may have to provide a good deal of specificity and even oversight in order to 
allay staff anxieties about "not doing the right thing". 

o Other Factors Affecting Implementation 

A number of staff and organizational factors can also play a role in the implementation 
process. Corwin (1983) notes that some changes can be introduced through staff 
inadvertence - these he calls "slippages". Kettner et al. (1990) identify a phenomenon 
called "drift" which can occur when, in carrying out a program, the staff gradually adapt 
the program to better meet their own needs and in so doing may move away from the 
needs of clientele. Both authors serve to remind us that the question of "when is a 
program up and running" has no final answer. It is only through regular monitoring of the 
program that one can speak to its fidelity of implementation. 

Up to this point we have not taken much note of the fact that this program is being or will 
be implemented as part of an ongoing organization that sponsors many other programs 
as well. Scheirer (1987) identifies six organizational components which can affect the 
extent of implementation. They are: 

o Nature of the Program - how much of the developmental work is done before 
implementation as opposed to its being done during or a* part of 
implementation? 

o Qualities of Clients - do the intended clients have the necessary prerequisites 
to benefit from the program (e.g. educational background & training, willingness 
to participate, transportation, etc.). 

o Characteristics of Program Deliverers - are there individual differences among 
staff members (e.g. training, education, values, attitudes, motivation for change, 
outdated notions of how the program should work,prior negative experience re 
management support & follow-through) that may impede implementation? Or, are 
there substantial costs in the change-over process (e.g. new procedures, 
technology, worker relationships or overload, devaluation of expertise, etc.) that 
may impede implementation? 

o Operating Aspects of Work Units - is the new program compatible with 
established ways of doing things or is a substantial change-over required? Are 
the informal work norms (informal standards of behavior) and supervisors 
supportive of the new program? 

o Overall Organizational Structure - does the organization as a whole support 
the new program ( e.g top-level backing and operational participation in decision 
making, adequate resources for equipment, space, training, etc.)? 

o Environmental Pressures - are there forces in the environment that can create 
instabilities for the program &/or staff ( e.g. unstable funding sources, regulating 
agencies, competing priorities, community opposition, conflicting values among 
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different organizational levels, etc.)? 

Such concerns will also form part of the LCGT's deliberations but may not be supported 
by formal reporting or information. More likely such concerns may appear through informal 
means or networks. As an information source this might be called "sensing" since it 
represents their "sense of the situation/organization" that they bring to their deliberations 
at that time. 

The functioning of the LCGT to determine the status of program implementation is 
portrayed in Figure 1 1 .2. 

The matter of determining effects from early returns is discussed in the next chapter. It 
could just as appropriately be discussed in this chapter since monitoring for intended 
effects and monitoring for program implementation can play back and forth on one 
another - the lack of intended effects leading to program changes and the subsequent 
monitoring of these changes to see if they are put in place and then, with what effects. 
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12. Maintaining and improving the Program 



This chapter is concerned with sustaining the intended effects of the program and 
identifying and infusing into it practices that will enhance efficiency &/or effectiveness. In 
this phase the Life Cycle Guidance Team (LCGT) continues to perform oversight 
functions but in contrast to the prior phase, its' focus now is concerned with intended 
effects or enhanced effects rather than just implementation. First however, the use of 
early returns as a monitoring device for intended effects is examined. 

12.1 What Do "Early Returns" Teii Us Concerning Clientele Benefits? 

As noted earlier, if there is a graduated rate of implementation then some information 
can be obtained early on about clientele benefits from the sites that are the first ones to 
implement the program. Such information should enable the LCGT to determine whether 
or not the program is on target in terms of intended effects. If the intended effects are 
occurring and further monitoring of successive sites shows that this trend is continuing, 
then the program can be considered "up & running" and the next stage can be entered. 
However, if the intended effects are not occurring then the LCGT has to make some 
diagnostic decisions about why they are "off-base" and what remedial steps need to be 
taken. The bases for these diagnostic decisions are the barriers that are encountered 
(either anticipated or not) and what has been or might be done about them. For this 
aspect of implementation, the barriers must be occurring in association with Main Events 
10-13 in the illustrative Program Logic Model (viz. from Provide Educational Experiences 
to KASA Change to Behavior/Practice Change to Consequences). To enable Diagnostic 
Decisions to be made and Corrective Actions to be taken, information needs to be 
collected from clientele on what they perceive the barriers to be for them, in a form such 
that they (the barriers) can be classified into these Main Events. This may dictate a 
specific kind of format or probing that elicits responses for the classification system. 
Examples of these are: 

Barriers Diagnostic Decisions Corrective Actions 

Some systematic Examine program content Recalibrate &/or restructure 

segment of the & process for specifics 



target audience 
(e.g. most needy, 
less affluent, 
minority, non- 
English speaking, 
etc.) is not com- 
pleting the program. 



that are involved in 
"dropping-ouf ; talk 
with "drop-outs". 



Tailor efforts & materials 
to specific needs 



Some systematic 
segment of the 
target audience 



Examine program content 
& process for specifics 
that create learning/ 



Recalibrate &/or restructure 



Tailor efforts & materials 



12-1 

224 



(e.g. most needy, practice problems; to specific needs 

less affluent, talk with those affected, 

minority, non- 
Engiish speaking, 
etc.) is not bene- 
fitting (showing 
KASAB change) as 
much as the others. 



12.2 Full Scale Implementation & Monitoring for Intended Effects 

Once the program can be considered "up & running" (fully implemented) the LCGT will 
give greater focus to monitoring for intended effects. The following forms of information 
will provide much of the basis for its' deliberations: 

o Clientele Benefits - results of the 3rd party surveys of clientele will be forwarded 
in a form that allows aggregation across units and the calculation of unit profiles 

o Successful Practices - staff will identify practices that they consider unusually 
effective/efficient or ineffective/inefficient in such a manner that individual, team 
or unit identities are not revealed. These practices can be forwarded to the 
LCGT for their analysis and synthesis, if necessary through a means that 
provides anonymity for the units as well as the staff members responding. Such 
results can be used to infuse new practices into the program as well as to 
provide guidance for staff training needs. They can also be used to identify 
structural problems in the program that may impede effective performance (e.g. 
mis-match of staff & client compatibilities, use of definitions/policies/practices that 
are exclusionary, etc.). 

o Success Stories - units will provide brief narrative descriptions to the LCGT of 
: - what they feel are examples of successes their staff has had in working with 
clientele, providing the program to new kinds of clientele.etc. These will usually 
focus on individual clientele, will be shared with other units and some will be 
included in the program and site newsletters. 

o Stakeholder Feedback - those stakeholders (or a sample thereof) who have 
been recipients of the newsletters) will be contacted (via phone or face-to-face) 
to ascertain how the newsletter &/or contact with those associated with the 
program (providers &/or recipients) has helped them to understand the nature 
and possible effects of the program and, solicit their overall impressions as to 
how they feel the program is being provided. 

* 

Figure 12.1 portrays the process that the LCGT would go through. Again ft is important 
to note the explicit provision of feedback to all of the program units as well as the sensing 
that takes place. The Diagnostic Decisions and Corrective Actions that the LCGT might 
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Table 12.1 Issues Addressed by the Life Cycle Guidance Team 
on Program Performance for the Different Kinds of Feedback 



Successful Practices | 


C i { ente 1 e Benef i t s 


HOW UNUSUAL ARE THE PRACTICES IDENTIFIED 
WITH REGARD TO EFFECTIVENESS OR 
EFF IC IENC Y? 

IS IT POSSIBLE TO INFUSE SUCH PRACTICES INTO 
OR EXTINGUISH THEM FROH THE PROGRAM? 

IF SO, WHAT WOULD BE REQUIRED AND WITH 
WHAT EFFECTS? 


WHAT DIAGNOSTIC D E C I S I ON S WERE MADE? 

WHAT CORRECTIVE AC T I ON S WERE TAKEN 
AND WITH WHAT EFFECT? 

WHAT STATEMENTS CAN BE MADE CONCERNING 
CLIENTELE BENEFITS? 


Success Stories 


Stakeholder Views 


DO THE STORIES REFLECT UNIQUE, 
PERSONALIZED ACCOUNTS OF HOW 
CLIENTELE B E N E F I T FROH THE PROGRAM? 

ARE THESE STORIES TESTIMONIALS FROM 
CLIENTS THEMSELVES? 

HOW MIGHT THESE STORIES BE USED TO 
-COMMUNICATE PROGRAM 
ACCOMPLISHMENTS? 


HAVE THEY RECEIVED THE NEWSLETTER? 

HAVE THEY READ THE NEWSLETTER & IF SO, 
HAVE THEY FOUND IT USEFUL IN DEVELOPING 
THEIR UNDERSTANDING OF THE PROGRAM? 

HAVE THEY RECEIVED PERSONAL CONTACTS 
FROM PROGRAM STAFF OR CLIENTELE & IF SO, HOW USEFUL 
HAVE THEY FOUND THEM IN DEVELOPING 
THEIR UNDERSTANDING OF THE PROGRAM? 

HOW ACCURATELY DO THEY PE RC E I VE 
WHAT THE PROGRAM IS TRYING TO ACCOMPLISH? 

WHAT ARE THE GAPS IN THEIR UNDERSTANDING 
AND HOW MIGHT THEY 3E RECONC I LED? 
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Figure 12.1 Forms of Feedback Provided to The Life-Cycle 
Guidance Team to Determine Status of Intended Effects 



YES THE PROGRAM 
IS 

"UP & RUNNING" 
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DELIBERATIONS 
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GUIDANCE | 
TEAM 




DIAGNOSE PROBLEMS 
MAKE CORRECTIONS 
t MONITOR CHANGES 



NO 




PROVIDE FEEDBACK 
& GUIDANCE 
TO UNITS 




NO 



CONTINUE TO 
MONITOR FOR 
INTENDED EFFECTS 
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make are described under "early returns". 



The deliberations of the LCGT may take different forms depending upon the class of 
feedback information that they are dealing with. Some possible forms of deliberation are 
given in Table 12.2. 

As a result of their deliberations the LCGT will develop ideas about the performance of 
the program and of the staff as well as how both might be improved. The group will also 
want to develop some incentives to encourage the adoption of exemplary practices if the 
decision is made to infuse them into the program. 

It is apparent from the preceding discussion that only through continued monitoring can 
the program be regarded as implemented so as to yield clientele effects. 

In the next chapter we examine different types of program redirection and their rationales. 
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13. Redirecting the Program 



All too often programs' candidacy for redirection are closely tied to variations in funding 
cycles and funding levels. When times are good the sensitivities to redirection may not 
be great. But, when funds become increasingly scarce the sensitivities to redirection are 
greatly heightened and it is not uncommon for reductions to be made in whatever is most 
convenient, at that particular point in time. Such actions may be unfortunate for both 
clientele and the organization in the longer run and may not reflect the kinds of decisions 
that would have been, or should have been made had less of a "crisis mentality" 
prevailed. In this chapter we outline some procedures that may be followed to arrive at 
a more orderly and thoughtful way to deal with program redirection,. 



13.1 An Hierarchical Framework for an Organization 

One can readily encounter a great deal of confusion concerning the use and nature of the 
relationship among such concepts as mission, goals, strategies, programs, objectives, etc. 
What should be called what and how do they relate to one another? Figure 1 3.1 presents 
one way of organizing and thinking about these concepts. A vision for the organization 
contains a short statement of how the organization will function effectively in the future 
viz. what it would like to become (Bryson, 1990). A mission statement is a justification for 
the current existence of an organization (Bryson, 1990). For what purpose or purposes 
does the organization exist? What societal needs does it fulfill? Bryson (1990) notes that 
clarifying and codifying a mission statement can help reduce conflict in an organization, 
aids discussion and helps to channel efforts in productive ways. Such a statement can 
also serve as a source of inspiration to employees and other stakeholders. Accompanying 
the mission statement is often found a statement of the organization's philosophy and 
values usually articulating how they regard clientele, other organizations and one another. 
Goals are broad ends towards which efforts are directed. They may at times be numerical 
in nature, e.g. by the year 2000, 75% of all farmland will be subject to Integrated Pest 
Management procedures.They may also be oriented towards the vision or serve as 
bridging mechanisms for moving the organization in that direction. Strategies are the 
broad areas of effort put forth to achieve these goals. Programs are specifically directed 
efforts that make up the strategies. Policies are listed with programs since they often play 
back and forth on one another. For example, it may be the policy of an educational 
organization to not provide any services unless they comprise a necessary part of an 
educational program. Objectives can be thought of as the next level of detail in a program 
while activities are the means by which objectives are attained, propelled of course by the 
necessary resources. Activities most often form part of a program. However, there may 
be times when they stand alone. For example, an educational organization may have a 
marketing activity which in itself does not have any educational purpose other than to 
maintain the visibility of the organization. 

All of the levels in this hierarchy are profoundly affected by emerging problems, issues 
or concerns (called EPIC'S for short). EPIC's can transform the vision or mission and can 
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Figure 13.1 An Hierarchical 

Framework for an Organization 




shrink or expand the organization in a variety of ways. As we shall see later in this 
chapter they can also have a profound effect on program redirection - just what gets 
redirected and in what ways. We shall also see that redirections are often made at the 
activity level as well as at the program level. 

13.2 Why & How Are Programs Redirected? 

Programs can be redirected in a variety of different ways and for a number of different 
reasons. They can be redesigned, resulting in consolidation or expansion, perhaps with 
new functions added or old ones deleted. Or, they can be phased-out completely so that 
they cease to exist after a designated point in time. Another option, not often mentioned, 
is that they can be transferred to the sponsorship of another organization in whole or in 
part. An extreme case of this latter possibility is for the parent organization to create an 
organization which will assume sponsorship for the particular program. We shall consider 
examples of each in the sections that follow. 

But, one might ask, why would an organization want to redirect its program(s)? The first 
and most optimistic answer is that the problem or need has been ameliorated to such an 
extent that the program as currently constituted is no longer relevant. A second possibility 
is that other problems have arisen which are of greater priority than the one addressed 
by the current program. In the extreme case this latter possibility could lead to zero 
funding of the program and hence a total phase-out. A third possibility is that the mission 
of the organization has changed so that the program is no longer one the organization 
should be sponsoring. Alternatively, the program or certain portions of it may have 
evolved in ways that put it outside the mission of the organization. Similarly, programs 
can be expanded because they are doing an excellent job and there are still many 
clientele that need assistance. Or, a program can be a stunning failure and fully merit 
phase-out. 

In Table 13.1 we attempt to systematize the types of redirection and identify an 
organization's reasons for doing so. Inspection of this table shows that the reasons or 
rationales for program redirection can vary considerably. A common cause for a change 
in the level or nature of the effort is a shifting priority for the program. Three others that 
are closely interrelated focus on increasing efficiency, avoiding duplication or combining 
resources/expertise. Whether a program or parts of it are related to the current mission 
can be an important reason for getting rid of it by phasing it out or transferring it to others. 
The needs of clientele are an important reason for changing the program or expanding 
its' functions. If the needs of clientele are met when the problem is ameliorated, then this 
becomes an important reason too, for downsizing, consolidation or phase-out. 
Conspicuously absent from this list are concerns about budgets, funding or effectiveness. 
While failure can lead to phase-out, program success doesn't necessarily lead to program 
expansion. Indeed, since programs are expected to be successful - "to work" - such 
performance, by itself, would not be a basis for expansion. Funding concerns are not 
directly apparent because they affect the priority setting process which in turn forms the 
basis for many decisions concerning program redirection. Since the priority setting 
process is so important it behooves us to examine it in more detail. But first, let us 
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Table 13.1 Types of Program Redirection 

& Rationales 



Typi of* Rid I rtct ! on 



Ration 



Change in Level of Effort 
Phase-up 
Phase-down 
Phase-out 

Change in Nature of Effort 



Meet Needs of Greater 
n of CI ienteie 

Problem Ameliorated 
Decreasing Priority 

Problem Ameliorated 
Decreasing Priority 
Program Fai lure 
Not Part of Mission 



Expanded Volunteer Roles 



Narrow Focus & Intensify Efforts 



Meet Needs of Greater 
# of CI ienteie 
Increase Efficiency 

Increase Clientele Effects 



Broaden Focus & Dilute Efforts 



Serve More Clientele 



New Functions Added 



Consolidate With Other Programs 



Change in Sponsorship of Effort 



Meet Additional Needs 
of CI ienteie 
Increasing Priority 
Increase Efficiency 

Problem Ameliorated 
Decreasing Priority 
Increase Efficiency. 



Co-Sponsorship With Other Organizations 
Without Disengagement 

With Gradual Disengagement 

Transfer To Other Organizations 

Al I Functions With 

An Existing Organization 

Some Functions With 

An Existing Organization 

A I I Functions By 

Creating An Organization 



Some Functions By 

Creating An Organization 



Combine Resources/Expertise 
Avoid Dupl i cat ion 

Not Part of Mission 
Avoid Dupl i cat ion 



Not Part of Mission 
Avoid Dupl icat ion 

Not Part of Mission 
Avoid Dupl icat ion 

Not Part of Mission 
Decreasing Priority 

Not Part of Mission 
Decreasing Priority 



consider some actual examples of some of the types of redirections listed in Table 13.1. 



13.3 What Are Some Actual Examples of Program Redirection? 

Imagine an educational organization that provides a service as part of an educational 
program in order to facilitate clientele acquiring certain practices. However, once clientele 
have instituted these practices and the practices have become fairly routine, the 
organization no longer wants to continue providing the service. As a result, the 
organization gets someone else to provide the service even if they have to train a cadre 
of people to be the service providers. Three different examples arise out of work in 
Extension: soil testing; scouting for pest management; and, master gardeners. Let us 
consider each in turn. 

o Soil Testing 

In the earlier years of Extension work, the testing of soil samples in order to determine 
their composition relative to what was needed for certain crops ( e.g. acidity/alkalinity; 
nutrient & mineral levels, etc.) was often provided to farmers free of charge as an 
inducement for them to adopt soil testing practices and incorporate them in their cropping 
plans. Once adopted however, Extension no longer wanted to provide this as a service, 
even for a fee. Hence, such testing was turned over to a fee based laboratory (whether 
public or private). 

o Scouting for Integrated Pest Management (IPM) 

Integrated pest management entails the sampling of pest populations on crops to see if 
their concentration has exceeded a threshold value such that crop yields would be 
threatened. If they had not reached such a threshold then pesticides would not be applied 
thereby curtailing costs and avoiding the risk of polluting the air, ground &/or surface 
water. If yields were threatened however, pesticides would be applied in amounts 
appropriate for that concentration of pests, thereby avoiding excessive dosages. The 
determination of these thresholds is made by "scouting" a field using sampling techniques 
to determine what existing concentrations are. In the early years of IPM, Extension would 
often provide "scouting" services as part of its* program to get producers to adopt IPM 
techniques. However, once IPM practices were established, Extension turned such efforts 
over to private entrepreneurs - often training them so as to be able to provide the 
services. 

o Master Gardeners 

In California, the Master Gardener (MG) program was instituted in response to a dramatic 
increase in calls to the county office from new homeowners concerning a wide variety of 
home gardening problems (Smith, 1989a). Indeed, the volume of calls was so great that 
the county staff could not begin to handle them all, let alone perform their other duties. 
The MG concept entailed giving volunteers training in home horticulture techniques and 
practices. Each volunteer would receive a number of hours of training in return for which 
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they would provide a number of hours of service (e.g. 40 hours of training for 100 hours 
of service) consulting with homeowners about their gardening problems. Once the MG 
program was established the county agent's job changed dramatically from one of a 
technical consultant to homeowners to one of a manager of a large cadre of MG's. The 
program engendered a great deal of commitment from the volunteers with many staying 
long beyond their initial number of hours. As the program matured, it developed its own 
governance or oversight structure with the more experienced volunteers providing 
guidance to the less experienced, thereby easing up on some of the agents' involvement 
in supervision. The MG program also resulted in a number of interesting spinoffs. Social 
networks were formed. A Master Gardener Society v ~ funded with the society having 
an annual state-wide meeting that was well atten volunteers covering their own 

expenses. Some of the volunteers even went k .<ass for themselves as home 
horticulturists. 

13.4 How Is the Redirection of a Single Program Determined? 

Many factors both internal and external to a program can affect its future direction. 
Those that are internai to the program focus on the organization's mission, the current 
status of the problem that was the impetus for the program initially, and the performance 
of the program relative to the problem. Figure 13.1 attempts to portray these kinds of 
considerations. No matter what the performance of the program is, the question is raised 
as to whether or not ft is within the current mission of the organization. If the mission has 
changed or the program has evolved in ways that put ft outside the mission, then it 
becomes a candidate for phase-out or transfer-out to another organization. If ft is within 
the mission, then the focus is turned to the current status of the problem. The problem 
could have been eliminated, improved, remained the same or gotten worse. All of these 
statuses have implications for how the program might be changed or redirected 
Interpretation of the status of the problem can give specific guidance as to such changes 
Table 13.2 gives different interpretations of each status and the actions that might be 
taken. If the problem no longer exists then obviously there is no longer a need for the 
program. However, if the problem has been eliminated because of the program then 
activities may have to be put In place to sustain these changes even though many parts 
of the program or even the program itself, are no longer needed. Similar considerations 
apply if the problem has been amelioratediathe program's target areas. In this case one 
might want to move the program on to new target areas while providing some activities 
to sustain the changes in the areas being left behind. However, if the problem has been 
only mildly improved in the target areas, then some increased or expanded efforts might 
be considered. If the problem remains the same, even in the target areas, then the 
program may be a failure and should be phased-out. However, if other clientele needs 
are muting the program's effects or the level of effort is not sufficient, then one may want 
to add new functions to the program or put forth more effort. Similar considerations aoDlv 
if the problem has worsened. 

But who we might ask should perform these deliberations? 
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13.5 What Role Does the Life Cycle Guidance Team Play in Program Redirection? 

The Life Cycle Guidance Team (LCGT) has carefully nurtured the program up to this point 
and would be a natural to perform such deliberations subject to one proviso. If they are 
so emotionally attached to the program that they can no longer deliberate about it in an 
impartial manner, then the composition of the team would need to be altered to give it 
greater objectivity or the functions would have to be performed by a deliberative body that 
is not so involved with the program. If an organization periodically reviews all of its 
programs with a focus on their future direction, then many of these functions will be 
performed by the group that is formed to carry out this review. Concerns that would enter 
into their deliberations would focus on a variety of factors external to the program. Let's 
see what some of these might be. 

13.6 What Factors Affect the Redirection of All of an Organization's Programs? 

Imagine a deliberative body that is given the task of reviewing all of an organization's 
programs to decide what the future status of each might be. What kinds of concerns 
might it focus on in the course of its' deliberations? Let us assume that the group adopts 
a systematic set of procedures [ as specified by Lambur & Burtner(1993) based on the 
work of I.McMillan (1983)] to conduct this review, as follows: 

o A clear, up-to-date mission statement must be available. If such is not 
available then one must be developed. If one is available but is lacking in clarity or 
specificity then it needs to be revised and refined. It is recommended that key 
stakeholders to the organization be involved in this process. 

o A program/activity profile is developed. This profile involves the identification 
of the who, what, when, where and why for the program or activity.[Activities can be 
included too since not all of them are encompassed by programs.] 

oo Who is served (clientele) by whom (staff) and who can improve this process 
(stakeholders)? What steps are involved in carrying out the program or activity and with 
what costs? 

oo When during the year does the program or activity take place ( all year, 
seasonally, intermittently)? 

oo Where does the program or activity take place (county, district, state; fixed or 
mobile)? 

oo Why was the program or activity started (what problem was it/ is it addressing)? 

o Profiles are reviewed and consensus reached. The profiles are widely 
reviewed by staff and stakeholders with adjustments being made to reach consensus on 
a final set. These adjustments may entail combining, dividing or altering parts of a profile 
so as to reach agreement on a final set. 

o Profiles are ranked and targeted. Each profile is evaluated on three 
dimensions of: (1) program attractiveness; (2) competitive position; and, (3) alternative 
coverage, using the following kinds of criteria: 
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oo Program Attractiveness. This dimension reflects the degree to which the 
program is attractive to the organization as a basis for current and future programming. 
Criteria to be considered are: 

* congruence with mission * complementariness with other 

* mandate programs/activities 

* programmatic fit * clientele base 

* existing expertise & resources * volunteer appeal 

* support base (financial & * measurability of results 
political) * prevention vs. treatment 

* education vs. service 

oo Competitive Position. This dimension focuses on the extent to which the 
organization has superior potential over other organizations to carry out the 
program/activity. Criteria to be considered are: 

* delivery system * research skills 

* funding history * technical skills 

* "track" record * organizational skills 

* quality of program/activity 

oo Alternative Coverage. This dimension is concerned with the extent to which 
other organizations are involved in the delivery of the same or similar kinds of 
programs/activities. Criteria to be considered are: 

* coverage is low if there are no other large organizations or 
few small ones attempting similar efforts 

Once the ranking has been completed each profile can be classified into r -je of eight 
categories as indicated in Table 13.3. [The reader will note that the use of the criteria 
resulted in a profile being assigned a high(strong) or low(weak) value for each of the 
three dimensions.] The essential features of the eight categories are described in Table 
13.4. 



When al! of the profiles have been categorized those that are in the phase-out/transfer- 
out categories will need to be reviewed for possible constraints in doing so. For example, 
are there legal issues, public domain considerations, no existing organizations to transfer 
to (so one has to be created), high versus low expenditure features, etc. that have to be 
dealt with? 

o Results of the categorization are communicated to administration 
o Decisions are made concerning those to be eliminated 
o Decisions are communicated to the organization 

o Management of phase-out/transfer-out process recognizing the importance of two 
components: emotional - the personal or affective aspect; and, the technical - the details 
involved in doing so. 
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13.7 Who Should Guide the Categorization Process? 

Whether the preceding process or some other is used, the question remains as to who 
in the organization is q alified and should be engaged to perform these functions? 
Lambur & Burtner (1993) recommend that such a deliberative group be composed of 
about 8 persons who: represent various groups and levels within the organization; have 
a broad based view of the organization; are respected and trusted by others; have strong 
interpersonal, communication and group problem-solving skills; reflect a balance between 
those that are task-oriented and those concerned with individual and organizational well- 
being; have awareness of resources outside the organization; and, are diverse in ways 
that are of importance to the organization. If there are a number of Life Cycle Guidance 
Teams in operation then they too should have some representation on this group as well 
but there must be a balance so that no one has a particular advantage. 

This deliberative body is responsible for the profiles being prepared on each 
program/activity. Whether they actually do this work themselves or have others do it, they 
are responsible for upholding the "integrity" of the profiling process. For example, profiles 
should be prepared with information that is equally accurate and comparable across all 
programs/activities. The profiles should be completed in a consistent manner for all 
programs/activities, etc. Similarly, they may reach out to others to' assist in their 
deliberations about the targeting and ranking. However, all involved must apply the 
standards in a consistent manner across all programs, etc. 

Lambur & Burtner (1993) call such a deliberative group a 'Transition Management 
Team** (TMT) to distinguish the fact that by going through a process that results in "letting 
go" of certain programs the organization itself is inducing change that can be disruptive 
and even traumatic. They cast such a procedure in the context of organizational change 
and provide a variety of procedures for determining readiness for such a procedure as 
well as for phasing-out programs. To assist them in the targeting and ranking process the 
TMT may hold a workshop in which a large number of people, both internal and external 
to the organization, are invited to participate ( Weisbord calls this the "open systems" 
approach to organization redirection, 1987). Such a workshop may entail 2 days and 
involve up to 100 persons including staff and administrators, stakeholders and clientele. 
Following an overview of the process, workgroups of 5 to 10 people are formed (vaiied 
with respect to background) with the profiles being divided among the workgroups so'that 
each group gets an equal number that are diverse in content. The number of profiles may 
range from 50 to 200. Each group has a facilitator to lead its discussion of the profiles 
and each participant has an up-to-date copy of the mission statement. Once all of their 
profiles have been ranked they are taped to the wall under the heading of one of the eight 
categories into which they were classified. After all groups have finished ,a "walk-about" 
period is allowed during which time *ach participant can register their objection to the 
categorization by placing a coloreo not on the profile. Those profiles with dots are 
discussed among the entire assembly. The workgroup that did the categorization gives 
their reasons and then the persons who objected give their reasons. Profiles not receiving 
any dots are placed on a separate wall under their appropriate category while those 
receiving dots are taken off the wall, reshuffled and divided among the workgroups save 
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that no workgroup gets its earlier profiles. Another iteration or two is conducted until 
virtually ail of the profiles are categorized. It is likely that for some profiles agreement may 
not be attainable which may be due to different aspects of the activity being emphasized 
( e.g. education vs. service) or that the information provided was inadequate. Profiles 
from categories with implications for phase-out or transfer-out are divided equally among 
the workgroups who review ail comments and arrive at a recommendation that is then 
shared with thd full assembly. Although the work of the participants is completed upon 
completion of the targeting and ranking workshop, a great deal of work remains to be 
done by the TMT or its successors). For now divestment plans must be laid out in 
sufficient detail so that the programs/activities can be phased-out or transferred-out - 
tasks which may require considerable time and effort and which may involve others who 
were not part of the TMT. 
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14. Special Topics 

This chapter deals with a number of special topics that are concerned primarily with 
- different aspects of the modeling process or the development of the Educational 
Experience Profile (EEP). These include: using the matrix of educational effects to 
estimate expected levels of impact that the program will have on the target audiences; 
modeling 2 or more programs simultaneously; involving others through model verification; 
calibrating the model for different locales; and, using the model to develop individual and 
unit plans of work. In addition, programs as belief systems and their similarities and 
differences with projects are examined in more detail, the use of a theory of learning and 
stage/process models to guide the development of the EEP are reviewed; and, the use 
of performance incidents as a management tool is discussed. 

14.1 Statements and Systems of Probabilistic Beliefs 

In Chapter 1 we proposed that a program can be thought of as a theory that relates a set 
of organized activities with supporting resources to intended results. Further, we 
suggested that this theory can be thought of as a system of beliefs that are probabilistic 
in nature. Let us examine the kinds of statements that might be made about linking a 
single activity with an intended result or outcome and their implied probabilities (keeping 
in mind that the link may involve curricular materials, staff, level of effort or even 
stakeholders): 



Belief Statements re: an Activity — > Outcome Linkage implied Probability 

o In similar situations in the past, this has worked.. High 

o in my experience this has worked well High 

o Our best evidence suggests that this will work High 

o We believe that this will work Medium 

o We hope that this will work Low 

o We feel confident that thi ; will work Medium/Low 

o We feel certain that this vill work because High 

o This wont fail because Medium/High 

o This is bound to be a success because High 



Imagine now a program that is composed of dozens of these linkages each with its own 
belief statement and it becomes easier to see that this theory is comprised of a 
system of beliefs. It might be interesting to examine just what staff do in devising a 
program from the "belief systems" perspective. 

14.2 A Comparative Analysis of Programs and Projects 

Reference to standard dictionaries shows very few differences between the definitions of 
a "program" and a "project". Those differences that do exist attribute more sequencing 
or ordering of activities to "program" than to "project". However, both entail the notion of 
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plan and schedule. In the definition of "program" used in this manual [a theory ( which is 
in itself a system of beliefs that are probabilistic in nature) that relates a set of organized 
activities with supporting resources to intended results] the concept of project can be 
subsumed under that of "program". How then might "project" be considered as a specific 
type of program? Let us examine some dimensions along which they may differ: 

o Deliverables - programs rarely have a fixed deliverable whereas projects nearly 
always do. 

o Duration - projects usually exist over a shorter, specific time span than do 
programs and tend to be phased out once their deliverable has been accomplished. 
Programs, in contrast, tend to be cyclical in nature - coming up for review and renewal 
in a periodic manner, often associated with a budget or legislative cycle. 

o Problem-focused - programs tend to be focused on the amelioration of 
problems moreso than do projects. 

o Scope & Complexity - programs are usually of greater scope and complexity 
than are projects. However, there are notable exceptions. 

o Outreach - programs usually have a greater outreach (#'s reached) than do 
projects. However, again there are notable exceptions. 

o Containment - programs often encompass or fund projects whereas projects 
seldom encompass or fund programs although, they often contain sub-projects. 

o Innovativeness - projects can be and often are more innovative than are 
programs - programs may sponsor projects to balance competing interests thereby 
appeasing both the proponents and opponents of change. 

o Stakeholders - programs usually have a greater number and variety of 
stakeholders than do projects. Again, there are notable exceptions. 

o Constituency - programs usually have a larger and more active constituency 
than do projects. Again, there are notable exceptions. 

o Expected Effects - programs usually have greater expected effects than do 
projects - again with notable exceptions. 

Having made all of these comparisons ( and undoubtedly there are-more), what then can - 
we conclude about programs and projects. For a project we would qualify our definition 
so that a project becomes a specific type of program that has a specified deliverable 
within a given time period. Often project theorists (Kerzner, 1989) will also introduce 
notions of performance levels; budget levels; organizational structure, etc.. However, for 
definitional purposes (as contrasted with operational purposes) we have not found this 
to be necessary. 

14.3 Estimating the Level of Impact for Different Target Audiences Using the 
Matrix of Educational Effects 

The Virginia Water Quality program design team was attempting to develop a common 
conceptual framework in order to bring together some 1 9 disparate programs. Once the 
matrix of Educational Effects had been developed for the 13 target audiences involved, 
the group felt that it would be useful for future programming efforts to estimate the level 
of effect they thought the integrated program would have with these audiences (Lambur 
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and Stephenson, 1990). Through some discussion they reached a consensus on the 
H(igh), M(edium) and L(ow) ratings assigned to the different KASAB's in Table 14.1. 
Such, results can be then used as a guide to levels of effort in the program development 



Table 14.1 Virginia Water Quality Expected Level of Impact for Different 
Target Audiences 

Level of Impact 1 



TARfiFT AUDIFMOF 


K 


A 


s 


A 


B 

hi 


FARMERS- 


H 


H 


H 


H 


H 


HOMEOWNER- 


H 


H 


L 


L 


L 


YOUTH: 


H 


H 




H 


H 


MUNICIPAL MANAGERS: 


M 


M 


M 


M 


M 


INDUSTRIAL MANAGERS: 


L 


L 


L 


L 


L 


MILITARY COMMANDERS: 


L 


L 


L 


L 


L 


ENVIRONMENTAL GROUPS: 


M 


M 


M 


M 


M 


WATERMEN: 


L 


H 


L 


H 


L 


PUBLIC: 


H 


M 


L 


L 


L 


LOCAL GOVT OFFICIALS: 


H 


H 




H 


H 


DEVELOPERS: 


H 


L 


L 


L 


L 


STATE AND FEDERAL 
OFFICIALS: 


H 


H 


H 


H 


H 


AGRICHEMICAL 
SUPPLIERS: 


M 


M 


M 


M 


M 



1 The level of impact refers to both what Extension has done and what 
Extension is going to do in one planning cycle. 

H = High level of impact 
M = Medium level of impact 
L = Low level of impact 
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phase as well as the allocation of staff time in working with different audiences in the 
program implementation phase. 

14.4 Modeling Two or More Programs Simultaneously 

In Virginia their Water Quality design team tried to bring 19 programs into a common 
framework for future programming purposes (Lambur & Stephenson, 1 990). In that effort 
the team was not concerned with maintaining the identity of each of the programs and 
for that many programs it would have been an almost impossible task. However, a 
smaller number of programs might be modeled simultaneously yet retain the identity of 
each, if they have some common goals or objectives. For example, in Maine two 
programs were modeled simultaneously that dealt with maximizing the independence of 
the elderly and another two that dealt with the development of community leaders (Killam, 
1989;1990). Indeed, ft may be considered a desirable result of the process to have one 
model that can be used to depict a number of different programs if such is achievable. 
If not achievable, the resulting models may be fairly similar. More time is required to 
model multiple programs and the results are usually more complicated even if one Logic 
Model can be used for all of them. This is so because the functional components may be 
somewhat different, especially for resources. If the same Logic Model can be used then 
differences in the functional components can be indicated by stars, asterisks, different 
colors, etc. If different Logic Models are required then the results may be even more 
complex. As a "rule of thumb" one should try to make the same Logic Model serve until 
ft becomes apparent to the Facilitator and/or the group that one just wont do ft all. An 
experienced Facilitator may come to this realization before the group does. 

14.5 Alternatives in Model Verification and the Involvement of Others 

The model verification process may be thought of not only as a means for checking on 
the extent to which the model "fits" the experience of others who do or may carry out the 
program but also as a means of involving them in what the design team has produced. 
Models can be verified by: (1) explaining the model(s) to other program providers, 
determining the extent to which it is reflective of their experience and modifying it 
accordingly*; (2) querying others through some kind of a more structured interview or 
survey; and, (3) observingihe program(s) as it is actually carried out. Only the first 
alternative has been used to-date (which is also believed to be the most desirable) but 
in three somewhat different ways. In California's Master Gardener program and in 
Mississippi's 4-H program the uninitiated were introduced to the model (there were 3-4 
such persons in each case) by the Facilitator (rather than a workgroup member, to avoid 
possible undue influence) and were able to make modifications which although minor, 
proved to be extremely valuable. For example, a way to reduce attrition among Master 
Gardeners and an overlooked way in which Agents and Volunteers affect youth were 
identified. In the Louisiana 4-H program design (Richard & Miller, 1992) a verification 



* This can also be accomplished in part, by including a few new members in the second 
workshop session. 
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team of 6 agents and one administrator (who had not been part of the original team) was 
convened and introduced to the model by the Facilitator. They were able to relate to the 
model and its different aspects very readily. They made a number of constructive 
contributions which were incorporated into the model but did not result in substantial 
changes to it. In the Hawaii middle management 4-H program agents who had not been 
on the work group were introduced to the model and made substantial modifications to 
it. After this, potential volunteers were introduced to the then modified model and they 
In turn made modifications to it (note: this is also an interesting alternative to obtaining 
stakeholder viewpoints as described in the prior section). The resultant model and 
components might also serve as a means of introducing new staff or others to the nature 
of the program. 

14.6 Calibrating the Model for Different Locales 

We have seen how the design team comes to a common location and develops a model 
that is the product of their experiences. One might ask then how such efforts get tailored 
to the different locales in which they must operate. All of the modeling is sufficiently 
generic so that it can be calibrated appropriately to a number of different locales. For 
example, different levels can be specified for the effects indicators (e.g., x% of the target 
audience will be able to do thus and so) and different levels of effort can be specified for 
the Implementation activities. If the design moves into a developmental phase then many 
such concerns would be dealt with there. This topic is also closely related to the next 
topic of developing unit and individual plans of work. 

14.7 Developing Unit and Individual Plans of Work Keyed to Events and Activities 

Individual plans of work-the who will do what when-can be integrated into events and 
activities by entering the person(s) name(s) under the activity and specifying timelines in 
days, weeks or months. If the activities are too general they can be broken out into tasks 
and then identify the individual(s) involved. The preparation of unit plans of work can also 
be a step in the process that either precedes or follows the preparation of the individual 
plans. 

14.8 Meta- Analysis of the Properties of Program Logic Models 

Program Logic Models can be thought of as having a number of different properties such 
as the number of main events, the number of activities and indicators for main events, 
the number and nature of barriers, barrier reductions, intervening events and spin-offs 
associated with each main event and its location in the sequence of events, the absolute 
and proportional distribution of resources across events, etc. A kind of secondary or meta- 
analysis (Rosenthal, 1984; Schwandt and Halpem, 1938) can be made of these 
properties both within a program to gauge their frequency and severity ( e.g. how do the 
barriers change or stay the same as we work through pair-wise events in the Program 
Logic Model) as well as across programs to gauge their relative properties ( e.g is there 
a common proportional distribution of resources or spin-offs among main events within 



and across Program Logic Models). Such analyses can provide guidance to a variety of 
staff: administrators as to common or structural problems; program developers as to 
problems they might program to avoid; and, researchers as to problems for which further 
research might be useful. 

14.9 Social Cognitive Theory (SCT) as a Guide to the Development of the EEP 

Many theories of "learning" do not lend themselves to use in specific, real life situations 
either because they are too general or allow for only a limited segment of the different 
kinds of "learning" of which humans are capable. One notable exception is the work of 
Albert Bandura and his colleagues (Bandura, 1991; 1988; 1986; Sims & Lorenzi.1992) 
on what is currently called "social cognitive theory" (formerly social learning theory). This 
theory emphasizes the acquisition of knowledge through the cognitive processing of 
information. In the following pages we summarize three aspects of SCT that Bandura 
feels are applicable to practical circumstances such as training and motivation (Bandura, 
1988): (1) developing competencies through modeling ( a process in which people learn 
through observing the behaviors of others so that they can change their own behavior 
without having to have the experience directly themselves); (2) influencing perceived self- 
efficacy (a set of beliefs about one's own ability "to organize and execute courses of 
action required to attain designated types of performances" (Bandura, 1986)); and, the 
use of goal setting as a motivational device. 

o Developing Competencies Through Guided Mastery Modeling 

In this approach appropriate skills are modeled, then practiced under simulated conditions 
and then the observers (learners) are helped to apply the newly acquired skills in real life 
situations in ways that will enhance successful performance, as follows: 

1 . Complex skills are broken out into component sub-skills. Component sub-skills 
are then modeled via videotape in easily mastered steps that lend themselves to being 
combined in different ways for different purposes; using many brief examples.- Modeling 
is enhanced if there is perceived similarity between the model and the modelers. 

2. Simulated situations are provided so that the modelers can practice their skills. 
Corrective modeling is used as a form oHeedback in order to assist modelers to improve 
their proficiency on sub-skills that need further mastery. They are practiced until a 
desired level of proficiency and spontaneity are attained. 

3. Self directed successes are then sought through what is called "graded transfer". 
Newly acquired skills are first used in selected real life situations that are likely to produce 
success. Modelers describe their successes and critique where they ran into difficulties. 
As they gain greater proficiency and confidence they gradually take on more difficult real 
life situations. 
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o Enhancing Perceived Self-Efficacy (SE) 

Since perceived SE affects what people will and won't undertake and how much effort 
they may put forth, attempts to build and enhance one's sense of SE may be considered 
desirable. Perceived SE can be enhanced by providing success or mastery experiences, 
by observing the success experiences of others who are like oneself, by providing realistic 
encouragements and, by altering the physiological state and/or one's interpretation of it 
(e.g. reduce stress or reinterpret the situation so that it is no longer stressful). These 
enhancements are all governed by one's belief about their ability to control or change the 
situation they are in or the events being dealt with, if some degree of control or change 
is not feasible then a strong sense of SE will not come into play. 

o Enhancing Motivation and Self-Efficacy (SE) by Goal Setting 

Forethought enables people to guide and motivate their behavior using internal standards 
and to develop incentives to sustain their efforts. Future goals are attained by the setting 
of sub-goals that are closer in time and evaluating one's accomplishments of these sub- 
goals. Goals can have a strong motivational effect by providing one with a sense of 
purpose and direction. By successfully attaining challenging sub-goals one's sense of SE 
is also enhanced. Such accomplishments also create self-satisfaction and an increased 
interest in what one is doing.Goais that are dictated by some authority are not likely to 
be as motivating as those in. which one has had a role in their development or has 
developed themselves (save for possible life threatening situations). Goal setting with 
informative feedback on accomplishments has been shown to be more effective than the 
use of goals alone, feedback alone' or neither goals nor feedback 

There are certain characteristics of goals that determine whether or not they will be 
motivating: 

Definiteness - goals the are explicit can better serve as guides for performance 
and evaluation than those that are general. 

Challenge - there is an optimal level at which to set a goal so it enhances 
motivation; ones that are set too high can be disappointing while ones that are set too low 
may cause a loss of interest due to their ease of accomplishment. 

Proximity - goals distant in the future are not likely to be motivating unless they 
are broken down into sub-goals that are achievable in the shorter term. The latter are 
more likely to sustain efforts over time. 

In the development cycle the LCGT may want to use these principles as an aid in 
selecting the mechanisms, means and settings that will be used in providing the EEP as 
well as in the preparation of the curricular materials (as described in Chapter 10). 
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14.10 Stages and Processes Involved in the Ways People Change 



Is it useful to think that in the course of changing people may move through different 
stages with some change processes being more appropriate for certain stages than for 
others? The work of Prochaska and his colleagues (Prochaska, et al., 1 992) suggests that 
this is so. They find that whether self-initiated or professionally initiated, individuals in the 
course of modifying their addictive behaviors progress through the stages of: (1) 
precontempiation; (2) contemplation; (3) preparation; (4) action; and, (5) maintenance. 
Further, individuals may recycle through these stages several times before the behavior 
is terminated. They conceive of this recycling as a spiral in which the individual may be 
thought of as moving to a higher level closer to the point of termination. Let us take as 
an example the cessation of cigarette smoking (they have also included work with 
substance abuse - food, alcohol and drugs - in a variety of populations). In the 
precontempiation stage smokers have no intention of changing their behavior in the 
foreseeable future. Indeed, the individuals may be unaware or underaware of their 
problem(s). Precontemplators may wish to change but don't seriously consider doing so. 
Resistance to the recognition or modification cf a problem is characteristic of 
precontempiation. In the contemplation stage individuals are aware that a problem exists 
and are serious about doing something about it but have not yet made a commitment -to 
do so. Cigarette smokers may remain in this stage for years without doing anything 
further. Individuals in the contemplation phase tend to weigh the positive effects of 
eliminating the behavior against the amount of effort, energy and loss involved in 
overcoming the problem. The next stage, preparation, involves both intention and some 
small behavioral change - such as smoking five less cigarettes per day. Individuals in this 
stage have not yet reached a criterion of effective action (e.g. abstinence) but intend to 
take such action in the very near future (e.g next few days, weeks or months). In the 
action stage individuals successfully make the changes necessary to overcome their 
problem(s) (e.g own behavior, experiences, environment) and maintain this altered 
behavior from one day to six months. The cigarette smoker is now "off of cigarettes and 
may have rid his/her environment of items which might stimulate- or evoke smoking 
behaviors (e.g. ashtrays, cigarette lighters, odors of freshly lit cigarettes, etc.). The final 
stage, maintenance, is one in which people work at preventing relapse and securing the 
gains they made in the prior stage. This stage lasts from six months after the initial action 
to an indeterminate time in the future (e.g a lifetime) wherein individuals actively work to 
sustain the changes made and to avoid relapse. For example, former cigarette smokers 
might carry a mint in their pocket and pop one in their mouth whenever they get a craving 
to smoke. Also, they may actively avoid sitting around smokers and might even become 
involved in an anti-smoking campaign. Should relapse occur individuals may recycle back 
to any of the earlier stages. Most smokers (85%) recycle back to the preparation or 
contemplation stages while the remainder may go back to the precontempiation stage (at 
least for self-changers). 

The authors find that the amount of improvement individuals make in a behavior 
modification program (e.g. smoking cessation) depends upon the stage they are in prior 
to participating in the program. Also, when the program is not appropriately focused on 
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modification program (e.g. smoking cessation) depends upon the stage they are in prior 
to participating in the program. Also, when the program is not appropriately focused on 
the stage individuals are in, improvement is much less. Further, if the program is focused 
on one stage but potential recruits are in a different stage, recruitment rates may be 
exceedingly low (e.g. recruiting for a program that is based on the action stage when 
potential recruits are in the pre-awareness stage). Finally, the authors find that there are 
change processes that are more appropriate to some stages than to others (e.g. 
consciousness raising is appropriate in moving from pre-contemplation to contemplation 
whereas reinforcement management [rewards, contracts, etc.] and support relationships 
are more appropriate for moving from action to maintenance). We have attempted to 
summarize these concepts and their relationships in Table 14.2. 

The notion of stages and processes is also supported by the work of Rogers (1983) in 
dealing with how people make decisions concerning whether or not to adopt an 
innovation. Rogers provides evidence for five stages that he calls: (1) knowledge; (2) 
persuasion; (3) decision; (4) implementation; and, (5) confirmation. In the knowledge 
stage a person becomes aware of the innovation's existence and develops some 
understanding of how it works. In affecting this awareness the mass media and 
cosmopolite (those from outside the social system) communication channels play a major 
role while the individual characteristics of socio-economic background and personality 
variables play a role in shaping who are the "earliest" persons to engage in such 
behaviors. In the second stage, persuasion, the person develops an attitude toward the 
innovation which can be either favorable or unfavorable. The formation of their attitude 
is affected by certain attributes of the innovation itself such as its' advantage relative to 
what is already in use, its compatibility with current practices or norms, its complexity, 
whether it can be tried out or not and whether it can be observed or not. Local and 
interpersonal (face-to-face) communication channels play a greater role in shaping these 
attitudes while there may be cues-to-action (e.g. adoption of a contraceptive due to a 
pregnancy scare that lead directly to behavior change). In the third stage, decision, the 
person involves themself in activities that lead to a decision - either to adopt or not to 
adopt. Results of trials (done by self or others) as well as existence of incentives will play 
roles in shaping this decision. In the fourth stage, Implementation, the person puts the 
innovation into use - perhaps making modifications to it. Factors influencing the extent of 
modification or reinvention are: complexity- and difficulty; lack of adopter knowledge re: 
details; the innovation is only a general concept or is used to solve a wide range of 
problems; a local adopter may want to assert "pride of ownership" by making some 
modifications; and, a change agency may have played a role in the adoption/modification 
process. Finally, in the confirmation stage the person seeks information that is 
supportive of their decision in the third stage but they may reverse that decision if 
confronted with conflicting messages (. viz. to discontinue if adoption made or to adopt if 
the innovation was initially rejected). An innovation may be discontinued because there 
is a better replacement or because the person is dissatisfied with the innovation's 
performance. These concepts and their relationships are summarized in Table 14.3. 
Rogers also identifies stages that an organization may go through in the innovation 
process (Rogers, 1983) and suggests stages that are involved in the evolution of a 
communication network (Rogers & Kincaid, 1981). 
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be worthy of consideration when thinking about the development of the educational 
experience profile (called the EEP in Chapter 10) as well as some of its associated 
modules/activities such as creating awareness, recruiting participants, etc. In Table 14.4 
we have attempted to develop a set of hypothetical stages and processes between 
stages using the KASAB attributes as a way of illustrating how the development team 
might want to develop their own stage-process model. The nature of movement between 
stages would also be something they would decide (viz. can an individual jump stages 
or must one move through each stage in turn, etc.). 



14.11 Analyzing Exposure Rates for the EEP 

As noted in Chapter 10, a number of variables can be used to describe a learning or 
educational experience. Those for time focus on such concerns as frequency, recency, 
intensity and duration of the experience. Such concerns can be used tc structure or 
apportion the amount of time devoted to certain skills and/or subject matters (S's). Let us 
assume for purposes of illustration that there are 6 S's to be covered in 6 experiential 
sessions (E's) of 90 minutes duration each session. We can form a matrix as illustrated 
below where the row- jpresent the S's, the columns represent the E's with each cell 
containing the amou .: of time to be devoted to S in that E. 



Sessions (E's) 



Skills/ 
Subj.'s 


1 


2 


3 


4 


5 


6 


Total 
time 


S1 


I! 3Q,Vv 




: r.: : '-io 








70 


S2 


30 


' ; ts : .> 


10' 




5- 




70 


S3 






, o 


• -6m 


0 




60 


S4 


0 


30 


20 


20 


:^:.j26:0;,; 




110 


S5 




0 




p 30 




30 


120 


S6 




iol 


••,.: V ,, 20 ..;: 


30 ■ 


30 


/, ' 30 : 


110 


Total 
time 


90 


90 


90 


90 


90 


90 


540 



The entries in the rows will reflect the amount of time devoted to that S in each session. 
They are repeated in other sessions to allow for further coverage or, for practice or 
rehearsal as an aid to retention. Skills and knowledges that are prerequisites would be 
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introduced in the earlier sessions while those that build on the prerequisites would be 
introduced later after the lower order skills or subject matter had been learned. 

A simple matrix format like this lends itself readily to spread sheet analyses on a personal 
computer with much larger and more complicated layouts being possible. It can be used 
as a planning tool and as a tracking tool for both group and individual instruction. 

In the development, implementation and maintenance cycles the LCGT may want to 
consider the use of such a tool. 



14.12 A Theory of Performance Incidents 



Figure 14. 1 Distribution of Performance Incidents 



Critical 




Ineffective Effective 
Frequency of Occurrence 



In observing our fellow workers (those who work with us, for us or for whom we work) we 
can all recall some activities they carried out which were considered unusually effective 
or efficient. Similarly, we can recall behaviors that were unusually ineffective or inefficient. 
In contrast, the greater mass of work behaviors that we observe in our day-to-day work 
lives do not stand out in any particular way. Those behaviors which stand out in our views 
as being extremely different can be called incidents of critical performance, or, "critical 
Incidents" for short (Flanagan, 1 954).[ Figure 1 4.1 illustrates these concepts.] They are 
critical not just because they are different but because they made a difference in the way 
the work was done as well. 

u &3 



Critical incidents are obtained by asking people directly involved in doing or supervising 
the work to give specific actions or behaviors that they have actually observed. The 
examples are of two types: (1) effective Incidents - those that lead to significantly better 
than average accomplishment of a particular job, assignment, mission or responsibility; 
(2) Ineffective Incidents - those that lead to significant delay, mistakes, omissions, lack 
of accomplishments or obstacles to achievement of work. An individual critical incident 
is not an evaluation of a person. It is an observation of "what happened", what action 
took place, and what were its consequences. 

In the usual procedure for gathering incidents, one person contributes only a few. 
Typically s/he is asked to write up eight of them - two effective and two ineffective 
incidents involving job or technical competence and two of each kind involving working 
with people. Because the incidents are collected from many people - enough to provide 
a good sampling of the job or job family under consideration - it is comparatively easy to 
assemble a pool of several hundred or more, depending upon the size of the job family. 
At this time, certain information is also requested about the people involved in each 
incident, to help in carrying out analytic studies of aggregate data for management 
purposes; but complete Individual anonymity is assured. 

After the incidents have been collected they can be analyzed and categorized :n different 
ways*. Usually, the incidents are sorted into categories based upon their common 
content. Descriptors are developed for each category so that incidents can be easily and 
reliably categorized. Eight to twelve categories are usually identified. One side of each 
category will contain effective incidents while the other side will contain ineffective 
incidents. Examples of categories resulting from such analyses are: (1) performing tasks 
accurately; (2) taking responsibility and initiating action; (3) responding to need for extra 
effort; (4) cooperating with others; (5) getting along with others;- (6) planning and 
organizing work; (7) motivating subordinates; (8) training and developing subordinates; 
and, (9) maintaining communications (Mayeske et al., 1966). They will of course, vary 
with the particular job family under consideration (Mayeske, 1966). 



* The reliability and validity of the "critical incident technique" methodology has been 
examined and found to be satisfactory save for some concerns about interobserver 
reliability. However, the latter is a problem only for judging individual as opposed to 
aggregate performance (Ronan & Latham, 1974; Anderson & Nilson, 1964). Individual 
personnel performance is best judged in terms of the accomplishment of workplans rather 
than in terms of behavioral extremes. The accomplishment of work plans may entail many 
more performance categories than those of the extremes (Hahn, et al., 1979). In addition, 
a personnel appraisal system should have certain properties that an aggregate monitoring 
system need not be concerned with (Davis & Verma,1993; Stufflebeam, et al., 1988). 
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Once categories have been developt J and the pool of incidents have been reliably 
categorized, a variety of analyses and uses can be made of them. They can be counted 
to see which categories contain the greatest number, sorted by level of responsibility to 
see how they change as one moves from technical to supervisory responsibilities, 
tabulated to see if there is a relationship between the type of incident and the length of 
time elapsed, etc. They can also be analyzed to determine the overall strengths and 
weaknesses of an organization, to identify training needs and performance standards to 
assess the relevance of training, to identify problems in supervision, to assess the effects 
of organizational changes over time, to identify structural impediments to performance 
and, to determine the degree of agreement on what constitutes effective and ineffective 
performance (Mayeske et al., 1966; Mayeske, 1966; Mayeske & Glickman, 1964). 

As a management tool the Life Cycle Guidance Team (LCGT) may decide to use some 
form of performance incident methodology to assist in guiding the program. 
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Chapter 15.0 Life Cycle Program Evaluation 



This chapter focuses on an evaluation of the Life Cycle Program Management (LCPM) 
process outlined in the preceding chapters. The evaluation questions and issues are 
organized by each cycle with some brief discussion given to the methods that could be 
employed. Life Cycle Program Evaluation (LCPE) is discussed and defined. Roles and 
standards for program evaluators (PE's) and the work they do are outlined. Most of the 
judgments made by the PE are centered on the extent to which the fidelity of focus on 
the problem was maintained and the plausibility of the program was sustained. [Ordinarily 
concerns with fidelity of focus would form part of those for plausibility. However, it is so 
subject to threat that it is elevated to a status similar to that of plausibility.] Other 
concerns are with the quality and integrity of the entire process. 

The PE's role is portrayed as one of ast tssing the effects of management decisions on 
program performance as well as the bases for their decisions and the accuracy of the 
information used in making their decisions. Special efforts to more thoroughly and 
rigorously determine programmatic effects are undertaken only if it is determined that 
there is something worth examining and there is a special need and resources to do so. 
[ These concerns are dealt with in the next chapter.] 

15.1 What is Life Cycle Program Evaluation? 

In the preceding chapters we have attempted to outline basic components of the LCPM 
process. But, we might ask, what would an evaluation of the different life cycles focus on? 
In this section we shall outline an approach to LCPE that focuses on a set of evaluation 
questions with issues specific to each cycle. The possible methods to be used to address 
these issues are organized around the cycles. This three way classification can be 
thought of as generating a three dimensional rectangle, as depicted in Figure 15.1. 
Questions and issues appropriate for each cycle are given in Table 15.1. Examination 
of these issues shows that most of the traditional focus of program evaluators (PE's) has 
been on the determination of clientele benefits. These concerns pertain to one or two of 
the issues in cycle five if one includes consequences along with benefits. Another focus 
for PE's has been on program improvement which deals with yet another issue in cycle 
five. At times, PE's have also focused on program implementation which deals with one 
of the issues in cycle four. The remainder of the issues are ones that are dealt with 
tangentiaily if at all, by PE's yet they comprise the bulk of the evaluation issues for the 
LCPM process. We shall deal with these issues and methods in greater detail once the 
nature of program evaluation and the ways in which it is carried out have been examined. 
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Figure 15.1 Classification of Evaluation 
Questions, Issues & Methods by Program 

Life Cycles 
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Figure 15.2, Major Areas of Evaluation 
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Table 15.1 Evaluation Questions & Issues Associated 

With Program Life Cycles 



Pro 



ram L i f 
ycles 



Genera I 
Quest i on s 



Specific Issues 



1. Problem Finding 



Are the problem & 

its' solutlon(s) 
"credible"? 



How urns the problem identified; by whom; 
why did they engage in problem finding; 

jtSlffed? 6 ^ WaS the proposed so'"*' 00 
How "well" was the process carried out? 



2. Program Design 



Were fidelity of 
focus & plausib- 
ility sustained? 



How "wel " was the process carried out? 
How readily could an independent set of 

peers understand the product? 
How many modifications did they make 

to It? 

How many modifications & of what type 
had to be made to the desi^i in the 
subsequent cycle? 



3. Program Development 



Were fidelity of 
focus ft plausib- 
i I ity sustained? 



How "we 1 1 "mas the process carried out? 

What was the nature & extent of potential 
clientele and expert(s) involvement? 

How readily could a synthesis of the 
modular results be made? 

Wasthe expected time to reach develop- 
ment attained? 

How many modifications ft of what type 
had to be made in the subsequent cycle? 



4, Program Implementation 



5, Program Maintenance 
ft Improvement 



Were fidelities 
of:focus:& of 
implementation; 
Maintained? Was 
plausibility 
sustained? 



Were fidelity of 
focus ft plauslb- 
ility sustained? 



Whrt corrective actions were taken ft with 
what effects? 

Has implementation been attained? Was 
the expected time to reach implement- 
ation attained? 

How many modifications ft of what type had 
to be made in the subsequent cycle? 

corrective actions were taken ft with 
wnat effects? 

How do cl ientele benefit from the progrw? 
hjve «x«5P ary practices been 

identified ft infused Into the program 

ft with what effects? 
What are the consequences of the 

clientele benefits? 

Are there uanticlpated "spin-offs" & If 
so, what are their consequences? 



6. Redirection 



Were "integrity" & 
"credibility" 
sustained? 



How "wel I" was the process carried out? 
What mas the nature and extent of prog rat 

provider ft stakeholder involvement? 
How many candidates were identified 

for redirection? 

How "any candidates were redirected ft 
with what effects? 
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15.2 What Does Program Evaluation Entail? 

In the general field of Evaluation, as outlined by Scriven (1993), program evaluation is 
merely one of seven topical areas in which evaluation is practiced. The others are: 
product, personnel, performance, policy and proposal evaluations as outlined in Figure 
15.2. The seventh , called meta-evaluation , deals with an evaluation of the evaluations 
for a particular topic or topical area. [ Scriven (1993) also identifies other topics besides 
the "Big Seven" but we have chosen to restrict our focus to these seven*.] Still other 
topics such as research or theory can be readily handled under the "Big Seven". We may 
recall that a program in our definition is a theory (see Chapter 1). Hence, program 
evaluation can be thought of as the evaluation of a theory. Similarly, the evaluation of 
research as a body of knowledge can be thought of as falling into the program or product 
areas. Further, program monitoring and program reviews (in which an entire program or 
programmatic area is reviewed by a team of "external" experts) can be thought of as a 
less rigorous and perhaps less comprehensive form of program evaluation. [ Scriven 
(1993) calls monitoring a form of proto-evaluation.] Monitoring can be done for purposes 
of implementation or compliance (the latter sometimes called "auditing"). 

! n r.nf- pa9eS that follow we shaH see that the PE ' S « in the course of an evaluation of the 
LCPM process, may carry out not only a program evaluation but product, personnel, 
performance and policy evaluations as well. 

Program evaluation is also closely related to a still emerging discipline called 
organization development" (Sikes et. al, 1989; Weisbora\1987) and PE's may 
increasingly find themselves using many methods from that emerging discipline, as we 
saw in Chapter 13. Indeed, the utilization of results of program evaluations perhaps can 
beat be aided by organization development "experts" who help to establish and guide 
administrative mechanisms for change resulting from the evaluation. Figure 15.3 attempts 
to depict the relationship that may come to exist between these general disciplines and 
the sub-specialty of program evaluation. 

While we are dealing with the topic it may be prudent to dwell on what program 
evaluation is not. It is best to regard it as neither science nor research even though it is 
often portrayed as both. It is a sub-discipline that uses a number of methods that 
scientists use in doing research. PE's use such tools as part of a systematic, disciplined 
effort to obtain information about a program or programmatic area. To label such efforts ■ 
as science and/or research serves to put the emphasis on values and standards that mav I 



be unattainable or even serve as an impediment to what is needed ( e.g the researcher's 
insistence on a randomized design to follow youth into adulthood to determine which 



* Some of these are: intradisciplinary (evaluation of everything comprising a 
discipline); a range of efforts from literary criticism and real estate appraisal to 
quality control efforts in industry. 
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youth development programs are most successful in producing community leaders 
in spite of the fact that it is virtually impossible to maintain such a design over an 
extended period of time, not to mention the cost & length of time involved). Rather 
the emphasis should be placed on obtaining objective, independent information of 
a sufficient quality that will enable sound judgments to be made about program 
performance. 



Figure 15.3 Re 1 at i onsh i p of 
Program Evaluation to 
Evaluation & Organization 
Development 



EVALUATION 




ORGANIZATION 
DEVELOPMENT 
✓ \ 



PROGRAM 
EVALUAT- 
ION 



15.3 How Is Program Evaluation Practiced? 

There are at least three aspects to program evaluation - the practitioners, the work they 
do and, the settings in which they work*. Let us examine each of them in turn. 

* See House (1993) for a discussion of the social consequences of evaluation as a 
profession. 
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o Principles for Evaiuators 



The vast bulk of program evaluation is not done by professional evaiuators but rather is 
done by staff of the program's organization, staff of allied organizations or by clientele or 
stakeholders themselves. The evaluation may take the form of a monitoring visit by a 
supervisor or a team, may constitute a full program review by an outside team, or may 
entail the evaluation of the performance of individuals or teams that carry out the 
programs. 

Professional PE's are usually expected to bring a more formal, disciplined and 
independent view of the program than would program or Rrogram related staff. Since they 
are not indoctrinated about the program they are expected to ask many uninformed 
questions in order to develop their understanding of the program. Such questions can at 
times be very revealing. The American Evaluation Association (AEA) has drafted a set 
of guiding principles for evaiuators, given succinctly in Table 15.2. 

* Systematic inquiry - intended to insure that appropriate technical standards are 
maintained, that misleading information or conclusions be avoided and that results are 
communicated with sufficient accuracy and detail to allow a critique of the evaluation and 
its' shortcomings. 

* Competence - intended to ensure that evaiuators have the appropriate skills and 
experience and that they not undertake evaluations beyond their scope of expertise. 

* Integrity/Honesty - intended to ensure that evaiuators are clear and accurate 
with clients concerning an evaluation's cost, strengths, weaknesses, and uses, changes 
made while in process, stakeholder interests (including their own) in the evaluation, 
avoidance of misrepresentation of procedures and results, etc.. 

* Respect for People - intended to ensure that security, dignity and sense of self- 
worth of all those involved in or affected by the evaluation, be protected. 

< 

, * Responsibilities for General and Public Welfare - intended to ensure that 
evaiuators are sensitive to and take into account the range of interests and values that 
may be related to the welfare of the public. 

[These standards are subject to revision and approval by the AEA members before 
becoming final.] 
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Table 15.3 Standards for Evaluations of 
Educational Programs, Projects & Materials 



♦♦Utility ♦♦Propriety 



*♦ Feasibi I ity ♦♦ Accuracy 



ADAPTED FROM THE JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL E VA L U A T I ON ( S TUFF L EB E AM > E T Al-.1981> 
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o The Work Program Evaluators Do 

PE's can be found working in a variety of settings and are often called by many different 
titles. Program analysts, management analysts, evaluation specialists, research ( name 
of discipline ) are just a few of the examples of the titles they may carry. They may have 
disciplinary training at the doctoral level or beyond.or may have less than a college 
education. Few however, will have received formal academic training as evaluators. More 
likely they will have built upon their training in some disciplinary area such as Sociology 
Psychology, Economics, Education, Educational Research, Health Sciences, Public 
Administration, etc. and the situations they work in as well as the topics they work on may 
not be far removedjfrom that discipline. In order to provide some guidance to those 
working in the field of educational evaluation, a set of standards for the evaluation of 
educational programs, projects and materials were developed by a committee of 
educational evaluators. The four categories of standards they developed are given in 
Table 15.3. 

•Utility - intended to ensure that the practical information needs of different 
audiences for the evaluation are served. 

* Feasibility - intended to ensure that the evaluation will be practical, frugal and 
sensitive to the needs of different interest groups. 

* Propriety - intended to ensure that the evaluation will be conducted in a legal 
and ethical manner with consideration given to the welfare of those involved or affected 
by it. 

* Accuracy - intended to ensure that the evaluation will provide information that 
is technically adequate to judge the worth or merit of the object of the evaluation. 

A revision of these standards will become available in 1994. The new version defines a 
standard as "a principle mutually agreed to by people engaged in a professional. practice 
and which, if met, will enhance the quality and fairness of the professional practice" The 
revised standards, of which there are 30, are still subsumed under the four general 
categones in Table 15.3. Each- standard, contains guidelines for its application, common 
errors and illustrative cases (Evaluation Practice News, October, 1993). 

These same categories also apply to the evaluation of personnel systems however, the 
standards and supporting examples are different (Stufflebeam, et al., 1988). 

At times PE's may be found doing what might be termed "special studies". They are not 
necessarily evaluative in nature but are more often devised to provide an organization 
with some special kinds of information that is needed for some particular purpose and 
that is not otherwise available. Some of these studies may be designed to illustrate a 
particular strength that the organization has in dealing with a particular problem, type of 
clientele or delivery mechanism (e.g. water pollution, parent involvement or work with 
volunteers). 
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At times PE's may also function as facilitators of a planning and development process. 
The focus of these efforts may be an evaluation design of a program or the design and 
development of a program such as that outlined in preceding chapters. Such efforts are 
always collaborative and cooperative for they could not be successful otherwise. Although 
third party Stat.-, with no commitment on the part of the facilitator as to what has been 
or might be done is an advantage, the role played by the PE is more that of a "coach" 
than that of an independent observer. Consequently, we are less inclined to regard this 
as evaluation and more inclined to regard it as a developmental effort to which the PE's, 
by virtue of their training and experience can make a positive contribution. [Scriven (1991 ) 
calls this "pre-formative" evaluation. But, for the coaching type of relationship we prefer 
to avoid the use of the term evaluation even though the service provided by the PE, 
which can best be thought of as program design and development, may be invaluable.] 



oo A Practitioner's Creed and the "Great Debate" 

A great debate has been going on for some, years now over the appropriateness of 
different approaches and methods to use in program evaluation. The debate covers a 
broad range of issues from the nature of science and knowledge to the ultimate nature 
of reality and the philosophical correctness of different approaches/methods (Sechrest; 
et at., 1 993; Shadish.et al.,1 991 )."Schools of thought" have developed around this debate 
with one set of polar opposites (presumably) called "quantitative" and "qualitative". To 
the extent that this great debate introduces fresh ideas and new approaches into the 
field, it can be a source of enrichment. To the extent that it forces choices however, it can 
have a stultifying effect on the field. Although waxing or waning in evaluation circles 
depending upon which authors on reads (Sechrest, et al., 1993a; 1993b; Reichardt & 
Rallis, 1994) and waning in psychological circles (APA Monitor, November, 1993) the 
future is likely to see this debate evaporate. As more means become available to do both 
in the course of one study these approaches will likely not only strengthen and reinforce 
one another but will become so intertwined in one's thought and practice that the 
distinction is no longer a meaningful one. [ Renata Tesch, the well known qualitative 
analyst tells us that computer programs are now available to do cross-over analyses from 
qualitative to quantitative data and vice-verse as well as to analyze "co-occurrences" 
across a time dimension with qualitative data (1992).] 

Tie practitioner however, must keep an open and receptive mind to all new approaches 
p/acticing a kind of eclecticism. It is through such an orientation that a discipline as well 
as its' practitioners can flourish and grow. [As one practitioner puts it, "we cannot afford 
the arrogance of philosophical correctness" (Affholter, 1993).] 
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o The Situations In Which Program Evaluators Work 

It is a truism that "rarely, if ever, does one bite the hand that feeds them". It is also a 
truism that "when bitten, hands that feed can strike back with great force". These truisms 
apply to many aspects of life including program evaluation. It is a rare manager who 
willingly allows critical questions to be raised about his/her programs let alone permits 
answers to be obtained. Usually such questions are thrust upon them by external forces 
and if the PE is not also an external force, critical results are not likely to be forthcoming. 
PE's cannot afford io be dependent upon the favorableness of their results for their 
livelihood - yet they often are. This is so because organizations often employ PE's not 
only for purposes of accountability and program improvement but as a defensive 
mechanism as well. Evaluations are seen as being "safer or easier to mute if done by 
someone in their employ. Then too, if subject to a third party, external evaluation, one's 
own PE can defend the organization against the findings by providing sophisticated 
critiques of the methods and techniques employed as well as the assumptions made 
about the program in order to evaluate it. 

The rare manager who allows critical questions to be raised about programs as well as 
candid answers to be sought usually has some mechanisms by which they can guard 
against "joining those on the rolls of the unemployed". Scriven (1993) identifies one of 
these mechanisms as being an "early warning system". By this is meant that the manager 
insists that s/he be forewarned of any critical or negative results early on so that s/he can 
develop an appropriate and constructive response before the results gain notoriety. PE's 
can also employ such a mechanism and can also provide some balance to the report of 
their results, assuming of course that there are some good things to say about the 
program. As a general principle however, a PE's livelihood should never depend 
upon the favorableness of their results. 

o Towards a Theory of Evaluation 

In their recent book "Foundations of Program Evaluation" the authors (William Shadish, 
Thomas Cook and Laura Leviton) set forth five categories that they feel should be 
considered in a theory of "social" program evaluation. These categories are derived in 
part, from their review and critique of seven major evaluation theorists who have had 
considerable prominence in the field over the last thirty years (Scriven, Campbell, Weiss, 
Wholey, Stake, Cronbach, Rossi). Although their focus is on "social" programs the 
questions they pose to the reader can be recast so that they might have a broader 
applicability. Some of the programs provided by Extension might not be regarded as 
social in nature even though they can be regarded as having a social 
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Table 1 5.4 Five Components of a Theory of Evaluation and their Key Questions 
I. Programming 

(a) What problem(s) does this program address? 

(b) Can the program be improved? 

(c) If so, is it worth improving? 

(d) If not, what might be done with it? 



II. Use 

(a) How might I (the PE) make sure my results get used in a timely manner to help the program? 

(i) Do I want to do so? If so, why? 

(ii) If not, can the results of the evaluation I do be used in other ways? 



III. Valuing 

(a) Is this a "good" program? 

(i) What do I mean by "good"? 

(ii) What is the justification for my conclusion? 



IV. Knowledge Construction 

(a) In what sense do I "know" what I purport to have learned about the program? 
(0 How confident am I about my response in (a)? 
(ii) What gives me that confidence? 



V. Evaluation Practice 

(a) How can I narrow my options so as to do a feasible evaluation given the constraints under 
which I must operate (skills, time, resources) and the large number of alternatives possible? 

(b) How do I define my role? Am I an educator, methodological expert or judge of program worth? 

(c) Which questions of the many possible should I ask? 

(i) How might I go about answering them - which methods should I use? 



Adapted from Shadish, Cook & Leviton (1991) 



impact - if the runoff from a feedlot gets in your drinking vater the "social" impact 
becomes painfully apparent. They introduce their components to the reader via a series 
of questions that they feel program evaluators should be better able to address once they 
have read their book. These components and their associated questions are presented 
in Table 15.4. We shall see in this chapter and those that follow that in the course of 
carrying out an LCPE or an In-Depth evaluation study, most of these questions get 
answered by the PE in the sequence of steps set forth and procedures involved. 
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15.4 How Should Program Evaluation Be Defined for Life Cycle Program 
Management? 

In the LCPM perspective, a program can be evaluated in any one of its' cycles with the 
issues addressed being somewhat different for the different cycles. However, in focusing 
on the issues for a given cycle, the evaluation cannot overlook what was done in the 
preceding cycles and must, without fail, examine the nature of the problem the program 
purports to address. Provisionally then, we shall define Life Cycle Program Evaluation 
(LCPE) as "an assessment of the accomplishments, impacts and value of a 
program in redressing or ameliorating a problem" . Clearly, some of the components 
of this definition will be ( and should be) cycle dependent. For example, accomplishments 
refer to the early stages of the program (design, development and implementation) while 
impacts refer to the effects of the program on the participants and the consequences of 
such effects ( including unanticipated ones). Values come into play in the evaluation of 
each cycle and in program redirection. In the evaluation of each cycle they involve 
judgments about the "worth" of the effort. In addition, in the redirection cycle they have 
two other aspects - absolute and relative value. Absolute value refers to the worth of the 
program to the organization and to society while relative refers to the worth of the 
program compared to others that are being offered by itself or by other organizations. 
Such concerns coupled with program performance information (including costs and 
perhaps cost-effectiveness as well) lead to decisions about program phase-out, changes 
in sponsorship and other forms of redirection, as outlined in Chapter 13. 

Life Cycle Program Evaluations (LCPE) may be done for a variety of reasons. Often they 
are done to inform an administrative, funding or elective body about the performance of 
the program wherein such results may be incorporated into their deliberations. Or, they 
may be done on a more routine basis as part of an organization's ongoing program 
evaluation efforts, perhaps guided by legislative (sunset provisions), funding or planning 
cycle requirements, 

15.5 Evaluation Issues & Methods for Problem Finding 

There are three main "methods".-(if it is indeed appropriate to call them such) that are of 
particular relevance for this cycle: (1) social philosophy and ethics; (2) logic; and, (3) the 
techniques of the inquisitive but skeptical reporter*. Hopefully their relevance will become 
apparent in the discussion that follows. 

One cannot consider the problem seeking and finding phase without at first recognizing 
the general societal background in which they take place. There are at least three 
general concepts that can be used as a shorthand way of keeping this general 
background in mind. They are: 



* The latter two taken together may be akin to what Scriven calls "probative logic" 
(Scriven, 1991). 
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Figure 15.4 The Nature & Incidence of Problems 
Encountered by Societal Background Conditions 
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o The Zeitgeist - Webster's Dictionary (1963) teiis us that this term refers to the 
general intellectual, moral and cultural state of the era. We will include in this term the 
prevailing belief systems concerning: the universe and its origins; natural forces and their 
causes; diseases and their origins; the role of rational and irrational forces in influencing 
destiny; the nature of the human makeup including ethnic and gender differences, etc.. 
Clearly, the Zeitgeist can play an important role in determining which problems are 
recognized (or not), articulated (or not) and redressed (or not). 

o Social System Functioning - the functioning of the social system ( broadly 
defined to include economic, social and political forces) can have a profound influence 
on the nature and incidence of problems as well as their recognition, articulation and 
redress ( or failure to do soV 

o The Role of Gc merit - governments and governmental policies, whether 
through active involvemei benign neglect, can play an important role in problem 
finding, similar to that of the Zeitgeist. 

These three concepts are highly interactive in nature with one another as well as with 
their effects on the problem seeking and finding phase, as depicted in Figure 1 5.4. Let 
us consider a hypothetical example. Suppose that the Zeitgeist embraces a type of "social 
darwinism" in which individuals (or organizations) who prosper financially are seen as 
being "more fit" than those who are less prosperous and that "progress" is made through 
the efforts of those who are "more fit". Further, suppose that policies and programs of the 
government actually serve to enhance the status and efforts of the "more fit" and to 
discourage those of the "less fit" (Parenti, 1 988; Donahue.1 994). Finally, suppose that the 
economy takes a downturn with the result that the "less fit" are disenfranchised in a 
variety of ways (e.g. go out of business, lose employment, lose all possessions, etc.) with 
a resultant increase in divorce, alcoholism, family abuse, suicides, etc. Problem finders 
may encounter the latter increased incidences but merely regard them as the inevitable 
problems of the "less fit" in an evolutionary process. Or, they may encounter them with 
the recommendations that programs be put in place to ease the pain of economic 
transition for the "less fit" in what is also an evolutionary process. Alternatively, they could 
recommend that government policies be altered to no longer favor the n more fir or even 
to actually constrain them in ways that make them more "socially accountable" (e.g. foster 
certain kinds of support mechanisms - such as cooperatives and worker owned 
businesses (Wisman, 1991; Weisbord, 1987) - for the "less fit"). They too might 
recommend programs to "ease the pain" but t'ney would also be trying to get at some of 
the root causes of the pain. 

The great cultural anthropologist Ruth Benedict (1934) once wrote that "no one sees the 
world with pristine eyes". This is as true for problem seekers and finders as it is for 
anyone else. The "perceptual proclivities" that they bring to this phase will be greatly 
influenced by the aforementioned societal background factors as well as by their own 
personal background, including such obvious factors as: their socioeconomic and ethnic 
background, age, gender, training and experience, to the loss obvious ones of: at whose 
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behest they are undertaking the effort, what relationship they will have to the problem 
(viz. who will devise a solution and who stands to benefit from carrying it out - if anyone), 
what relationship they have to the organizations or groups involved, etc. In an effort to 
ferret out such "perceptual proclivities" as part of assessing the adequacy of this phase 
we proposed in Table 15.1 the 5W7H framework of the investigative reporter plus some 
judgments of a logical and technical nature concerning how "well" the process was carried 
out. The ways in which the problem was defined, solutions proposed and reasons for 
doing so, are also dealt with. Let us systematically examine each in turn: 

o Who Initiated the Problem Seeking Activity? 

The "who" is important because organizations or groups tend to search for problems for 
which they have a vested interest or capability. This vested interest can affect every 
aspect of the cycle - from information gathering to analysis to interpretation. The PE 
needs to trace out these relations! tips and determine whether or not this vested interest 
"swayed" the results in a certain direction. 

o Why Was the Problem Seeking Activity initiated? 

Was there a definite motive for the activity or was it part of a larger, generalized and 
periodic problem identification effort? Having a definite motive is not an indictment of 
anything or anybody. For example, an organization may say that "we have a genuine 
concern about the well-being of children in this country" and then set out to conduct (or 
commission someone else to conduct) a problem finding effort which documents the state 
of children in a completely objective and defensible manner. However, the PE needs to 
make a determination as to what the motives were and how they might have aff ected the 
results - if at all. 

c What Did the Problem Seeking Activity Entail? 

This question deals with the nature and scope of the effort. The PE will want to make 
judgments about the adequacy and credibility of the effort especially with reference to the 
problem(s) identified and the nature and level of effort put forth for their solution(s). Two 
key sub-questions are: from whom was the information obtained; and, by what means? 
Organizations may use advisory councils (Block, et.al, 1 992) or other means which do not 
entail a representative cross-section of persons who should be making contributions to 
the process. Even when a representative cross-section is attained the means of 
contribution (e.g written responses, personal appearance at a particular locale at a 
particular time) may exclude participation by many, thereby "swaying" the results. Or, 
alternatively, questions may have been formatted or posed in such a manner as to lead 
the responses/respondents in a particular direction. 
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o When Was the Problem Seeking Activity Conduu d? 



The "when" refers to how current the effort was and whether or not "datedness" should 
be an issue in judging whether or not the problem identified is still a problem. Or, 
alternatively, the problem may have since been redefined or is now viewed in a different 
way. 

o Where Was the Problem Seeking Activity Conducted? 

Were the efforts highly localized in nature or were they very general? The PE will want 
to make judgments about whether or not such concerns had an effect on the results. 



o How "Well" Was the Problem Seeking Activity Conducted? 

The PE will inevitably form some opinions about the "goodness" of the problem seeking 
effort, is it a "good" job (meaning well thought out and as thorough as desirable) or is it 
a "perfunctory" job that was really not worth the effortThen too, the "sophistication" of 
those carrying out the effort needs to be taken into account when making such 
judgments. Have the seekers had a great deal of prior experience and/or training or is ft 
something they did on a "shoestring" with little or no prior experience? 

o What Factors Influenced The Definition of The Problem? 

If those who gathered the information are different from those who analyze and/or 
interpret it, then the same 5W/H framework can be applied to the definition of the 
problem as a means of ferreting out possible influences. However, there are some 
broader concerns that should be dealt with first. These relate to judgments tho PE may 
be able to make as to whether or not the "societal background" factors or "perceptual 
proclivities" may have affected the recognition of the problem or the manner in which it 
was defined. Rogers (1983) notes that there is often a tendency to regard individuals as 
responsible for what are really system shortcomings (or malfunctions) and to regard the 
individuals as needing "fixing" with little or no regard for changing the system. The 
Zeitgeist and government policies (including benign neglect) may also support and 
reinforce this kind of viewpoint, per our earlier hypothetical example. 

o What Factors Influenced Tho Proposed Solution(s)? 

There are two aspects to proposed solutions: what is being proposed; and, why they are 
being proposed. Let us focus on the motivations first: 



15-16 



ERIC 



881 



oo Motivations for Proposed Solutions 

* Best Qualified - the first and most obvious reason for an organization 
proposing a solution is that they are the best qualified to do so. This may or may not be 
so and the PE needs to make some determination of the veracity of their claims, if 
indeed, any are made. 

* Gain Competitive Advantage - a second reason which may or may not 
be closely related to the first, is for the organization to gain a competitive advantage over 
some other organization or program, especially in an era of tight resources. Why the 
proposing organization did so is something the FE needs to determine. 



* Enhance Status With A Funding Source - an organization may offer a 
program primarily for purposes of appealing to a funding source that would otherwise be 
unavailable as a source of support. [Some organizations may even implement a program 
on a low priority topic to gain such support*.] 

* Contemporaneousness - an organization may offer a program because 
it needs to show that it is associated with issues or concerns that are contemporary in 
nature, sometimes to dispel an out-of date, "old fogey" image. 

* Need for Recognition - an individual or organization may offer a program 
because they need enhanced visibility. It is not uncommon for elected officials to ally 
themselves with a problem area and offer programs in that area as a means of enhancing 
their status in the community and/or with their constituents (this is not to deny that they 
may have genuine concerns for the topic or problem). So too may organizations need to 
increase their visibility to a variety of stakeholders or to the public in general. 

* No One Else To Do It - organizations may at times be "given" programs 
to carry out because there is no one else to turn to. In the case of elected officials there 
may not be anyone else under their jurisdiction and this is especially so for organizations 
located in remote or sparsely populated areas. 

Undoubtedly the PE will be able to think of other questions as well and is encouraged to 
do so. 



The opposite can also occur when a funding committee of elected officials forces a 
program on an agency so as to increase the appeal of their budget to the committee or 
to some of its' members. 
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oo What Was Proposed? 

The "what was proposed" becomes more problematic than the "why" for it is here that 
different approaches to the same problem have to be weighed and chosen among or 
packaged in amounts that are proportional to the different needs being addressed. Here 
the PE needs to focus on whether or not alternative approaches were even considered, 
how comprehensive they were, whether or not the level of effort to ameliorate the 
problem was at all realistic, whether or not there was collaboration or duplication of effort 
with 

other organizations, whether or to what extent the "societal background" factors or 
"perceptual proclivities" of the problem seekers/finders played a role in the proposed 
solution and if so, how?, whether or to what extent disciplinary biases came into play ( 
e.g. health versus education versus regulation; or, "I'll define my specialty as meeting the 
problem and hence be able to do what I like doing"); and, whether or to what extent the 
proposed solutions are based on the assumption that other, lower order needs will 
continue to be met [the maintenance (or met) needs becoming unmet of Scriven & Roth, 
1978.] 



Again the PE may think of yet other questions and is encouraged to do so. We will have 
more to say about some of these matters in sections that follow. 



15.6 Evaluation Issues and Methods for Program Design 

There are two main methods used to assess the issues for this cycle: (1 ) logic; and, (2) 
qualitative analysis of how the process was earned out and with what result(s). The 
qualitative analysis referred to here is not restricted to those familiar to the researcher 
(Patton,1990) but is more likely to entail a series of judgments concerning how well the 
process was carried out and whether or to what extent the design team may have 
"drifted" away from a direct focus on the problem, either intentionally or inadvertently 
(hence the emphasis on the small q). The twin concerns of fidelity of focus on the 
problem and plausibility of the design are the overarching ones to which an 
assessment of the adequacy of this cycle is oriented. Let us examine the "goodness" of 
this process through a series of questions and then use these answers to guide our 
judgments about fidelity of focus and plausibility. 

o How Well Was the Program Design Process Carried Out? 

This question can be broken down into a series of questions concerned with how the 
design team was formed, how they interacted with one another and with the facilitator, 
how stakeholder involvement was carried out end analyzed, how thorough the modeling 
was and how well it held up to the scrutiny of peers (if any) ana the development team, 
what the nature of the commitment to next steps was, and how this was facilitated by the 
executive briefing (if there was one) 
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oo What Was the Nature and Composition of the Design Team? 

The design team should have had experienced members, a preponderance of whom 
should have had directly relevant experience with the clientele-to-be of the program . The 
PE needs to determine if this was a "good" mix in terms of the design they developed. 
Whatever the mix was, the PE needs to determine whether or not the team members 
brought to their meetings pre-formed notions of how they would be working with clientele - 
a delivery system bias, if you will. Further, did such a bias - if one was found - lead to a 
weaker or less relevant design than might otherwise have been attained? Such concerns 
lead naturally to the next question concerning the nature of the team's interactions. 



oo What Was the Nature of the Team's Interactions? 

Some of the information needed to answer this question may not be readily available to 
the PE except by talking with some of the original participants, if they are still around. 
Attendance at all of the meetings was considered highly desirable and some attendance 
roster would indicate who were absentees and for which sessions. Such absences 
preclude their contributing to the process and if their's is a specialized viewpoint, then 
some aspects of the design may be deficient. More important however, is the nature of 
the team's interactions with one another. If one or two members were unduly vocal then 
their influence on the process may have been disproportionate to the quality of their 
contributions. The PE needs to determine whether the design was truly a team product 
or bore an excess of influence from a few members and how that influence affected the 
design. These questions also involve the skill of the facilitators) in guiding the team 
through the steps. If the facilitator(s) did a proper job, the question of undue influence 
would be tangential. 

oo How Was Stakeholder Involvement Obtained and Analyzed? 

The PE needs to ask: "was the nature of stakeholder involvement appropriate for the 
process and topic under consideration?". If individual interviews were conducted, were 
their results very revealing or might an alternative means such as focus groups, have been 
more useful? Whatever form the interviews took, was there a high rate of participation? 
Were the intended number of stakeholders interviewed? Did most of those who were to 
attend the focus groups actually do so? Were the analyses done in a systematic and 
thoughtful manner or were they rather perfunctory? The PE will likely think of other 
questions depending on the source documents that are available. 

oo How Complete and Representative Were the Modeling Results? 

For these concerns the PE will want to focus on whether all of the steps in the modeling 
were completed and if not, why not? Were relevant documents, if any, incorporated into 
the process? Do the results reflect any depth of thought or are they rather simplistic in 
nature? Could non-participant peers readily relate to the results or were elaborate 
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explanations/changes needed and if so, why? Did the development team in the next cycle 
have to make many modifications to the design and if so, why? 

oo What Was the Nature of the Commitment to Next Steps? 

Was anv commitment made by the executive or administrative council to make next steps 
and if so. whaTdid they involve (e.g. resources, policy changes, etc.)? Did the execut.ve 
briefing facilitate this commitment (if there was a briefing)? 

o Was Fidelity of Focus on the Problem Maintained? 

The main concern here is whether the design team stayed focused on the problem in 
carrvino out their design efforts or "drifted off course", either intentionally or 
unintentionally. If, in the judgment of the PE "drift" occurred, then the reasons for this 
need I to ' be ascertained (if possible). If intentional, then why? If unintentional, then what 
are the factors involved? What should be done with the design - can it be refocused or 
ould it be scrapped? 

o Was a Plausible Design Produced? 

If and only if fidelity was sustained, is it meaningful to ask about the plausibility of the 
design If plausibility questions are appropriate, then those given in Chapter 5 should be 
used to arrive at a judgment concerning the degree of plausibility of the design. 

15.7 Evaluation Issues and Methods for Program Development 

As with the previous cycle, the two main methods for assessing the adequacy of this 
cyde are : (1) logic; and, (2) qualitative analysis. A series of questions are answered 
concerning how "well" the program development process was earned out wrth the 
answers used to guide judgments concerning whether or not fidelity of focus on the 
problem had been preserved and, if so, whether a plausible program had been 
developed. 

o How "Well" Was the Process Carried Out? 

The most important qualitative question deals with the nature and extent of staff 
commitment to the developmental cycle. Other qualitative questions focus on the 
thoroughness of the work done for the different modules. A premium is placed on 
obtaining potential clientele input and expert opinion along the way, for through such 
means it is believed that fidelity of focus on the problem will be maintained. 

oo What Was the Nature and Extent of Commitment of Staff? 

It is not uncommon to find that staff have myriad responsibilities - even perhaps too much 
at times, with few opportunities for "sloughing or some of them. Another assignment 
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then, even if considered desirable would be an added burden to an already overcrowded 
schedule. Time management problems and feelings of stress could increase greatly. Add 
to this the fact that the team members may bring different disciplinary backgrounds to the 
work of the team and that they may be rewarded for successful work in their respective 
disciplinary areas, not in inter-disciplinary work. Also, profound personality differences 
may exist among the team members. Given these conditions the potential for conflict, 
delay and failure would be very great. In addition, the appropriateness of the staff 
assignments in terms of their competencies to contribute to the developmental cycle may 
be examined. The PE will want to determine to what extent these conditions were present 
and how they might have affected the program development process. 

oo How Adequate Was the Developmental Work for the Different Modules? 

The PE will examine the products resulting from each of the modules as well as the 
procedures used to develop them to arrive at some judgment about their quality. Quality 
criteria will involve the use of clientele input and expert opinion including program 
providers, plus the near absence of disciplinary "biases" (viz. some disciplines may 
predominate but for the proper reasons). The extent to which and ways in which the 
products could be integrated in the final task would also be a consideration here. 

o Was Fidelity of Focus on the Problem Maintained? 

The PE will use the results of the prior questions plus others s/he may have thought of 
to determine whether or not fidelity of focus was maintained. If not ihen there is no point 
in proceeding further and that is so for all negative responses to the fidelity and 
plausibility questions that follow. If yes, then the next question becomes: 

o Was a Plausible Program Developed? 

To answer this question the PE will want to apply the criteria given in Chapter 1 1 as well 
as to revisit those on plausibility given in Chapter 5 and used for the prior cycle. 



15.8 Evaluation Issues & Methods for Program Implementation 

In this cycle the LCGT decided to implement the program and the program has 
presumably reached a stage of implementation where it becomes meaningful to ask about 
problems encountered in implementation and how their resolution may have affected the 
program. It also becomes meaningful to ask about the extent of implementation and 
whether the program that has been implemented is the one intended. Answers to these 
questions will, as in previous cycles, help guide judgments that the PE will make 
concerning plausibility of the implemented program and fidelity. However, fidelity takes 
on two different meanings in this cycle - one deals with fidelity of focus on the problem - 
while the other deals with fidelity of implementation. Since the LCGT plays a key role in 
overseeing implementation, some attention will be given to their functioning . Since much 
of their decisionmaking depends on the various forms of feedback (unit accomplishment 
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reports, hot-line queries, supervisory site-visit reports, mentor site reports), their veracity 
would be examined as well. Hence in addition to logic and qualitative analysis many of 
the other tools commonly used in program evaluation will also come into play. These 
include sampling, structured survey and interview techniques and their analysis, etc. [ We 
will not deal in any detail with these methods since there are a variety of books that 
already do this. See for example, Henry (1990) for sampling, Patton (1990) for qualitative 
and Rossi & Freeman (1993 ) for quantitative approaches, Krueger (1988) for focus 
groups, etc.] Veracity checks of the different sources of information will also enable the 
PE to make judgments about the nature of the program that is in operation. 

o What Problems Were Encountered/Resolved and With What Effects? 

The PE will want to examine the proceedings from the LCGTs meetings to ascertain the 
barriers that were encountered, the diagnostic decisions that were made and the 
corrective actions that were taken. From these plus such other information that is 
available and relevant the PE will try to tease out what the possible implications of these 
corrective actions were for both fidelity of focus on the problem and fidelity of 
implementation. The PE may want to talk with field staff and clientele in the course of 
these deliberations. If both fidelities are maintained then and only then does it become 
meaningful to question the veracity of the various forms of feedback, if fidelities aren't 
maintained then questions need to be raised about the reason for existence of the 
program as is and whether it should be redirected somehow. 

o How Dependable Are the Various Forms of Feedback? 

Depending upon how much the LCGT used the various forms of feedback in their 
deliberations, the PE will want to derive a means of checking on their veracity. The 
means will likely be composed of a mix of methods including interviews with a sample of 
the sources providing the feedback to judge the quality of the information that they have 
provided. If the quality of the information is such that it is misleading , then the PE will 
want to make some remedial recommendations and take these shortcomings into account 
in judging plausibility. If the quality is acceotable then the PE will want to arrive at some 
final judgments concerning the two kinds of fidelities and plausibility. It should follow that 
if the two kinds of fidelities have been maintained then so too has plausibility been 
sustained. 



15.9 Evaluation issues and Methods for Program Maintenance and Improvement 

Given that the program was implemented as intended, the LCGT will have shifted its' 
focus to monitoring for intended effects. For these concerns the LCGT relies on the 
forms of feedback concerning clientele benefits, successful practices, success stories and 
stakeholder v. s, plus such other information of a less formal nature thU they bring to 
the team setting. Jefore focusing on the veracity of these sources however, the PE will 
examine the kinds of barriers that were encountered, diagnostic decisions made and 
corrective actions taken, and ascertain their implications for the maintenance/improvement 
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of the program. For both kinds of concerns logic and qualitative analysis will come into 
play. However, some of the more traditional program evaluation tools may be used when 
examining the veracity of the information. As in the preceding cycle, these tools might 
include sampling, structured survey and interview techniques and their analysis, etc. 

o What Problems Were Encountered/Resolved and With What Effects? 

The PE will deal with this question in a manner similar to that of the prior cycie. However, 
the emphasis will be not only on fidelity of focus but on the sustenance of plausibility as 
well. If these are found to be upheld then it becomes meaningful to examine the various 
forms of feedback. If they have not been upheld then concerns with refocusing or 
redirection need to be raised. 

o How Dependable Are the Various Forms of Feedback? 

The most critical form of feedback concerns the 3rd party surveys of the clientele benefits 
for they are the source for gauging whether or not intended effects are occurring. If they 
are inaccurate, then so too are the LCGT's decisions about intended effects. The PE will 
want to devise a means to check on their accuracy, assuming of course that they are 
done properly (as specified in Chapter 12). If not done properly, then the PE will want 
to conduct his/her own assessment, which may involve client interviews using some 
sampling plan, etc. 

The next important sources of feedback involve success stories and successful practices. 
The success stories give a clear, concise picture of the benefits of the program to a single 
individual client. Their credibility should be checked if such is deemed necessary and 
appropriate. If new practices are infused into the program then the ways in which they are 
identified, introduced into the program and "tracked" for their success rate will need to be 
examined to see if they meet reasonable criteria of successful performance ( or other 
such criteria in the case of efficiency). Finally, the success of the effort to keep 
stakeholders involved will be examined to determine how well it was done and whether 
or not some independent interviews need to be conducted to document their veracity. The 
PE may want to use the questions in Table 7.1 as a guide for probing stakeholder 
understanding. 

The level of effort devoted ta these verification exercises will depend very much on the 
size and scope of the program and available resources as well as on how well 
management has done its' job. 

The PE will then make some final judgments concerning the maintenance of fidelity and 
sustenance of plausibility. If the lack of veracity of the forms of feedback put either of 
these in doubt, then the PE will want to make some recommendations concerning 
remedial or redirective actions that might be taken. 
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An alternative to the above would be to conduct some form of field investigation involving 
clientele, that would be completely independent of the rhetoric of the program, 
reporting system, the staff, etc. This might be akin to what Scriven calls "goal free" 
evaluation (1993, 1991). Of course the investigatory staff would have to have an 
appropriate level of naivete' concerning the program. 

15.10 Evaluation Issues and Methods for Program Redirection 

Chapter 1 3 showed how a program can be a candidate for redirection either individually 
or with respect to all of the programs that the organization offers. Whether such 
judgments are made for a single program by the LCGT or by a larger group for the whole 
organization, certain considerations must be dealt with. They deal with the organization's 
mission and the current status of the problem. Programs that are no longer within the 
organization's mission are automatically candidates for redirection. So too it may be for 
a problem that has changed or for a program that has failed to impact the problem. The 
PE will want to check to see if there is a mission statement that serves adequately as 
a guide to redirection and whether the nature of the problem(s) have been dealt with in 
any redirections that have taken place. However, the redirection process for a total 
organization is a much more complicated process involving a wide variety of others - it 
is more difficult to do and more prone to controversy. Basically however, the methods to 
be used are logic and qualitative analysis perhaps with some interviews with former 
participants to gauge their retrospective views and reactions. 

The PE will want to focus on how the targeting and ranking process was carried out and 
whether judgments were corrupted by: factors or special interests; "chunking" (a program 
profile that is prepared in such a way that no components or activities can be eliminated); 
erroneous classifications; how well the phase-out or transfer out process was managed, 
etc, in order to judge the integrity of the process. 

15.11 An Algorithm for Ufe Cycle Program Evaluators 

Suppose that you are a PE and that you are requested to conduct a third party, 
independent evaluation of a particular program. Suppose further that you choose to 
function in a fairly independent manner so as to avoid being unduly influenced by the 
staff. Before you can design an evaluation however you will need to learn something 
about what stage or cycle the program is in as well as what transpired in earlier cycles. 
A recommended way of doing this is to start at the first cycle (problem finding) and 
systematically work through the evaluative questions for each cycle until one reaches a 
point where one of two conditions prevails: (1) further effort would be fruitless because 
of what was found out; or (2) more information is needed and would be useful but a 
greater level of effort may be required to obtain it. For example, if a field information 
gathering effort is required then some considerable additional amount of resources may 
be needed over and above the time of the one or two PE's who have been assessing the 
adequacy of the various cycles. [Alternatively, their time augmented by travel and 
transcription expenses might suffice.] 



An algorithm to guide this process is given in Figure 15.5. It shows the PE as proceeding 
to the next cycle only if the questions are answered appropriately for the current cycle. 
If not, then the PE need not proceed further but is to make some recommendations about 
how the situation might be remedied (including being scrapped). [A PE might conceivably 
examine what was or is being done in other cycles in order to strengthen these 
recommendations.] We have not depicted a large field data collection in conjunction with 
the implementation cycle because one may not be needed - a small sample effort might 
suffice to determine the veracity of the forms of feedback. Program effects are 
investigated only if the current and preceding cycle conditions have been met and some 
formal, independent effort is warranted. [Otherwise, for normal management operations, 
the information that is available will suffice.] For this a more substantial effort may be 
required and hence, is discussed in the next chapter. 

It may be of interest to speculate on a few of the kinds of conditions or decisions that 
would affect the recommendations a PE might make. There may be events that impinge 
on the LCGT over which they have little or no control. For example, budget cuts may 
result in staff reductions in ways that no longer allow a sufficient level of effort to be 
devoted to the program - hence a loss of plausibility. The PE might recommend that the 
program be restructured so as to be concentrated at an appropriate level of effort either 
by working more intensively in fewer target areas or with fewer clientele. Short of being 
able to make such changes, the PE might recommend phase-out. Another is that in the 
time since the program was started a better understanding of the nature of the problem 
has developed which suggests certain changes be made in the program. These could 
entail restructuring, revision of materials and delivery methods or phase-out. 

Decisions made by the LCGT which would affect recommendations concern 
accomodations and compromises that were made in different cycles that would cause 
"drift" to occur. For example, in moving to new target areas changes in scheduling in a 
locale may cause them to move away from the "neediest" clientele. Hence, the PE would 
recommend some sort of refocusing. 

Finally, upon examination of the reporting system the PE may find that the numbers 
reported tend to be "biased" in a direction that is considered desirable. The PE may 
determine the extent of this "bias" and recommend some corrective actions that could be 
taken by the LCGT ( e.g. closer supervision of the sources of bias). 

15.12 What Role Do Stakeholders Play in the LCPE Process? 

In the preceding chapters we have dealt extensively with stakeholder involvement in the 
LCPM process yet have dealt only slightly with the concept in the LCPE process. This is 
because much of the work that PE's would do would not require much stakeholder 
involvement. If all of the conditions of stakeholder involvement have been met as part of 
the program, there should be little doubt as to who they are or what their interest in the 
program might be. It is not necessary to reach out to a large number of them to seek their 
views concerning the performance of the program. Their views should be known through 
the feedback they provide and which the PE's substantiate. However, the PE's might want 
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to set up an advisory group composed of key stakeholders that they work with and review 
progress with them in the course of doing their investigation(s), especially if some 
extensive fiGld work is involved. For the latter, some additional effort might be made to 
gather the views of a larger number of stakeholders, perhaps using some of the questions 
in Table 7.1 as a guide. 

Of far more critical concern for the PE(s) is who is requesting the LCPE and what their 
reasons are for doing so. The reasons may arise out of conflicts that are intransigent in 
nature among different groups and are pursued with a passion that can be perilous for 
whoever gets in the way. Such a situation might be a compelling reason for having the 
advisory group cited above. 
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Chapter 16.0 Conducting In-Depth Evaluation Studies 



In this chapter we will assume that there is a compelling need to evaluate an ongoing 
activity of an organization but that this activity may or may not be classifiable as a 
program. We shall call this a programmatic effort/topic. Further, we will assume that 
this need overrides any considerations that may have resulted from the steps gone 
through in Chapter 15, if such steps were actually taken. This need may arise from 
legislation or some other form of mandate, may result from an organization's need to 
document its capabilities or assess its' potentialities or, may be an absolute condition of 
funding. 

There are three main purposes for conducting in-depth evaluation studies: 

o Program Justification - to document that the program resources are being used 
in a responsible way for the intended purposes and that these purposes are being 
achieved. 

o Program Implementation/Improvement - to document that the program is 
being carried out in the manner intended and/or to identify ways in which program 
performance in terms of delivery, efficiency and/or effectiveness might be enhanced. 

o Policy Formulation - to assess the possible effects of changes in the way 
activities are currently conducted or to assess the potential of new ventures. 

Conceivably, a single study might satisfy all three purposes. However, it is more common 
for a single study to focus on one of the purposes as its' primary purpose. In addition to 
these three purposes, in-depth studies may be initiated for strategic or tactical 
purposes as well. Examples of these are: 

*To inform, postpone or avoid a debate * To demonstrate capability 

*To lend credence, support &/or consolation to a * To gain recognition 

minority viewpoint e.g. the group that was * To demonstrate relevance 

outvoted) * To enhance status 

*To fend off "threats" (actual or potential) with a funding source 
*To gain a competitive advantage 

The first two often emanate from legislative committees while the remainder are ones that 
an organization or program director may pursue and are similar to some of those cited 
in the previous chapter as an organization's reasons for sponsoring a program. 

In Extension, in the past 15 years, a variety of national evaluation studies have been 
conducted. Some of them focused on existing programs such as Integrated Pest 
Management, Nutrition Education for Low Income Families, 4-H Alumni and Youth-at- 
Risk. Other studies focused on certain delivery methods within Extension such as 
Leadership Development and Volunteerism - topics which cut across most Extension 
programs. A third kind of study focused on a variety of policy concerns including the 
functioning of Cooperative Extension as a system or as part of a larger extension- 



16-1 

294 




16-2 



Li - 

O; 
CM 




co to 

C Q g 
Q) 5fc Q) 

Hi 



Q> 



o 

I 

2 

•o 
c 
CO 

I 



52 a 

§8 



£ 

-S3 



cB § 

.S y> 

S S to 

S £ o 



Q) 
C 
Q> 

2 <* 

C Q) 

o ^ 

Q) % 

££ 



1 

•2 
c 

E 
E 
§ 



Q) 

* § 

w co 

0) *c 
w *- 

0) CA 



lis 8 18 

(OA* 
CD >« 

CD Q) s - 



<0 



2 § 

£ S 

II 

18 

q> c 
££ 



1 5 

lis 

§«* | 

^ 52 ° 

Q> -* 

5 J3 

o o 8 



c -2 
? « o 

Q> CO 5 
CO g CD 

s * s 
% t § 



v» CO 



CD 

co .c 

* a 

£ £ 



*o ^ CO 

III 

£ <* co 



a £ -8 

O Q O 

as 

0) 0) Q\ 

8 g 8* 




S JR S 5 
P CD CO ^ 

S o o co 



as » 
Ml 

o c 5 



«8 

o ^ 
co 

*- c 
2 o 
£ *^ 

CO 

§ § 

^ *- 

t3 C 



CO 

2 

2 § 
S 5 



s 2 

£ * 



co 2 

s § 

31 



s § 

£ * 

o ,© 

C«fc 
— 0) 



8 S 

CC vl Q 

S. E 
g> <& o 
co co 

is 

CO C 




*s o. 

co ^5 >3 

e o § 

— CO i5 

2 53 0) 
c co c 



CO CO 



co i5 

Q> O 
CO CO 



c: 
0) 

a §£ 

co o 5 



16-3 



E 
2 

si 

3 CO 



5 1 1 a I 
121ft 

CO t q ^ 5 • 

<5 2 to k c 



I 

c cd 

II 



52 

6 «3 

ll 

5 2 
|§ 



CD 



5> 



2 -® 



CO 

CD 
CO 



2 
co C 



.0) 

•5 ^ € 

J? & iS 
§>iS co 

11 si 



00 

ii 

5 5l co 

518 



CO 

5 O 
is ^ 

S3 CO 
2UJ 
oO 

5 © 



«2 

§1.8 

co O 



S 



1 = 

s s 

8.8. 

" E 
o 

c c 

55 



s 

iu 
o 



2 o> 



§ 



a* •* 

4 «* «8 

•c 4= *C 



p 
p 



- 1 a 

8 £ §■ 

o i * 

CO 

ii* 

J CD 42 



4S € I 

© «o 5 

^ S £ 

3: co Q) 

4C co s 



CD 

is e 

0) 

8 EE 

#1 

II* 

V £ is 

0) O O 
co O co 



c eg 

0) £ 
CO O) 

a s 
s ^ 

Is 

§ 5 « 
*c -p 3* 



c 

c 
c 

CO 

E 

& % 

o c 

c £ 

£ «> 
u. O) 

^ CO 

>* C 

H 



5 

I 



CO 4= 



S 

CD *P CO 



II 

r 

! 



° s 

S c > 



l fe 

-I 

2 E 



CO 

c 

s 

Uj 

a 

co 

U. co 



CO 

1 

Q. 

E 

* m 

i 

3 
C 

73 
C 

8 



C 



El 



o r 

g § 
S E 



^ CD Q) 
C t £ © 



> CQ cq CD 
rC i5 t > 



111 
1 6 | 



9> 



sic; 

§ s 45 45 



O 

"co 

I 
a 

CO 

5 



U. to 



S 

IS 



CO r- 

S .2 

it 
II 

a. q 



5 c 

"p Q> 

"5 E 
is ^ 

a ^ 

CO s 



16-4 




go- 2 

f 

K a o co 
£ v) o w 

0) CO 



E "t © qj 
ctj 2 4^ c 

CO C 



cd 
o co 

Si* 

-a c ° 
^ to w 

CO 



CD 

-C* CD 
^ C <o | 

^ * c 2 

o5 £ Q. © 2 

0 . «- « 3S 

S c to S o 

1 S "o § s 



5 

g 

SI* 

CD ^ CO 
^ CO .CD 

O S) ^ 

E * g 

lis 

*C <o ^ 



f of 

til! 



00 ^ -s 



m vj kj. 
< ^ w to 



US 

*Q CD CO 
CtJ ^ £ 

£ co 
=> S eg 

o cd a 



CtJ 

c o S 

I ° TO 

5 *" 05 

5*1 

co P — 

ill 



O) CO 

I £ 

It 
A c 

C Q> 



CO 



CO 



a: «* 



£tf CD 

S S CD 

co co 5: 
o 00 q> 

§ °> £ 

*• w a 

w a. o 

it* 

CD *S ' 

s a 

CD O 
T3 SsO 



I 3 I 

3 *- tg h- 

I if * 
ills 

lege 

C £Z co 
"O E O CD 



^ Xf CD 0 !Z 
CD CD S 1 

Q.Q V, to £ 

SIMS 

*» y CO CD 

ex ^ c co *5 



§• 

o 

§1 

o 2 

co «0 
§ S 

CD ^ 

o co 



p <0 

4s TO 

1 1 

*CD 

CD m 

C CO 
CD 

CD CD 



fl 

t! 

11 



O "5 JQ 



CD 





s 9- a a "g 

§.-8 TO 

§*to c § 



E is 
S co 

* CD 

Is 



C CD 

IS 

c co 



CD 



If 



8 K 
I li 

« «: a 



c 

co O 5 



16-5 




X" 

o 
07 



ERIC 



16-6 



CO 



I 

E S 

sis 
ill 
II- 

J I 05 
«r> -j- m 
«C SO 



1 
I 

S 



5= 

a 

CO Q 
2? 5 

is 

5l 
II 
IS 



c 

c 

g Q) Q) 

° c S> 

^aa 

c o 
» $ « 

a « § 



a 
§ 

•s-a £ 

fc Q> * 



(0 

SP 

s 

co 3= 
a 

a a 

c § 

Q) 

O CO 



11 

■s* 

« £ 



2 a 

§ c a 
&a * 

i«i 

CO O CO 



2 03 

a s 



^ E ° 

£83 

CO 55 

to c 

eo * q> 
(0 0% 



£>a 

ST -5 

C Q> 

■8 w 

5 § 

ts 



I • .9 
2s *«• *s 

||1 

p O CO 

V. W mm 





CO 



, , **: 

I a a » a 

a n » * w 
o i2 •c « *. 

-nit 




a § «e § 

O 



CO g ojj 

I si § 




Q) 

*f s 



til* 

§a 



5 



8 a 



5 



a a Q. O 



<* g ^ 
Q> £ * 

| co * 
CO C 

£ Q> Cb 

a co 



- ° a 

CO s *5 

f «* 8a 

a to 10 

8 • 5 2 
5 58 S •? 

C £ ^ A 




list 

Si *t 

ill I 




a to 



a 

0) 



§1 

a s a 

co o 5 



16-7 



CO 

lit 
o 



CO 

c 
2 



o 

8 



? 

9 

5 



9 

3> 

C 
9 

m 

I 

z 



2 



o £ 

If 

w o 

CO Q. 



CO 

E 
E 

CO 

II 



! 

OH * 

S co 
c o 

CO .*?-* 

*^ 

o 8 

II 
* § 

£ E 



° g 

S3 ~ 

CO 

•0 g 

if 

•i ^ 

fZ co 

§ E 



si $ 
ill 

811 

Hi 
III 




8 

O Q) 



55 m S 



J8 5 ^ 

■g ag 
jo ,c 

S S 
o a a 



^1 I 



2 * 
6 



Q> 

-til 

§£ 8 2 

EgiSil 

Sao.* 



to 

Is 
II 

Q O 

C C 
Q> « 



1**3 



CO CL 



CO co 



JO 



est 

*2 CO *fc 

CO CO co 

^ ft 

I 

c 

CO 

E 

E 
o 

CJ 



?i 

Q> CO 
CO 

Hi 

IS 



T3 

a 

I 

Oft 



g 

o 

C m 

R 1 
o o 



.o 

I § 

E .9- 
c co 



*5 

a 

CO 

c: 
£ 

, o 

U. Ui 



CO 



CO 

o 
CD 

o 

I 

CO 
QC 

CO 

ct 

CO 

o 

CO 
Ui 
O 



« S 

5 S 

o E 

■c co 

2 § 

co £ 



o co € 



£|§ 

CO co 5 

8 § I 

CO CQ Q) 



•5 

^ s ^ 
g s a Q) | 



! 



•0 to 



££ 



ill** 
If |I 1 




I 
I 

CD 

§ 

CD 

E 



I 

CQ 



frl* 

s a « 

n Q ^ 
ft to 



.* E 



5 



32 



^T: 75 S CO *r ^ 



I 

s 

Q> 

f 

E 

CD 



Ml 

o 3 .. 

S I 

•I 8f 

111 

Pi 



"5 3 
El 



i 



CO 

£ 

CO 

i 
$ 

I 

CD 



f 
o 

c: 

0) 



rj co t: 

co 5 a 

2* k CH 
O CD «S 
C «C CO 

E CO 
_ § o 

o * S 
« CD * 

t 5 f 
1/3 ^ | 

(BOS 

13 CO * 

C S io 

111 
ill 

111 



lip 

c -5 5 oi ft 
m ^ "a fe 



2g«ti| 

-"si's 

x co a w 

1 ? cB 
^ 5 a «fe cn 



I 

CO 

o 

a 

CO 

a 

CJ 

s 

il 

ft i2 

i "> 
o v> 

o» to 

IS* 

£ c 
OS 

c 5 

i| 

co o 

IS 
II 



o 
CO 



c 

CD 

tM 

£ o £ 
to o 5 



ERIC 



16-8 



1 

CO 



i s 

Ui oc 



I* 

Q> 



TO CO to 



CO 



CO 

o 

— OQ -Q 

CO ^ 
Ui 5T 

CO 

q> C 
C ^ £ 



to o 

Q. co 

83 

g- ° 

o <b 

° s 

co <o 
c o 

Q> CO 

co o 



0 to §• o 

1 Ha 

So? . 

5 v. u c 
a ^ £ 

« * S CO 
9> ^ CD 
O) C e fO 



C 
O 

_ 2 
5 

0)0)0 

i M 



5 g| s 



a "5 §> 



si 



§■5 
5 



4 

! 

0)0 



2 
co 

Q> 

E 

o ^ 
co £ 



to 



£ C CD 



■si 



§•1 



c 

.$0 Q 

CO v, 

c g 

"O CO 



CO 

o 



CO 

>% 
a 

1 
! 



3 

c 
c 

8 

CO 
CO 



f 



5 

to 

CD 



CD CO o *5 

5 III 

Q> co «3> 

O E o>| 
£ § co "t5 
K co o ^3 



V> i CD 



«1 



Q CD 



MM 



CD T3 — 

CO ^ P 

C C Q) 

o 2 



CD T3 
co 5 ° 

CD fc* *D 
^ co ^ 

£ g co 

$ $ 

SSI 

co •§ 

Q. CD O 

Ui 



CO 

3 CO 

P CD 

S co 
go 

E co 

CQ CD 

CD £ 

S ° 

co £ 

CD O 

a to 

to to 



0> 

s 

CO 



co 
E 
E 



CO c: 

s I 
af 

Q. Q 



16-9 



CM 




Is 



si 

Q) £ 

i § 

§^ 
11 



v. 

J3 




o 

it 



w • - S 



§.co 

co 



3 



II 
12 



i 



IT* 

a a -a 

* S 5 

£ si 



Q) O) 
C CQ 



s 

co a 

S3 

s s 

§§ 

X c 
0) -Q 



- 3 - "3 



I 1 " 
|« 

§ 2"*= 

o s a 



58 

s 

I 

Q) 



i 



Q> 

<& e 

CD <0 

£ a 

11 



"fc -9 
<2 * § 

W Q) < 

ii* 

1 



1 1 1 

§s § I 
a. s -8 — 

(D it Ul -Q 

C — 




o ^ 

\Q.2 

f *i 
co O 5 



16-10 



OS 



research complex. Brief descriptions of some of these studies are given in Tables 16.1*, 
16.2* and 16.3*. The first 8 studies focus on the impacts of different programmatic efforts 
(Table 16.1). The next 4 deal with implementation concerns (Table 16.2) while the last 
6 focus more on system concerns with a policy orientation (Table 16.3). The initiation of 
such studies assumes that there is something of value to be learned from what is going 
on that is of sufficient magnitude to justify the expenditure of considerable time ( perhaps 
as much as 1-5 years) and resources ( usually in the range of 50,000 to 750,000 dollars). 
These studies often have some requirement that the principal investigator or PE have 
some third party status with respect to the program or topic being investigated. 

16.1 Comparative Analysis of In-Depth National Evaluation Studies 

Let us examine each of these types of studies in turn to see if there are some general 
principles that might be deduced as guidelines for future efforts. The 8 impact studies all 
address how clientele of the programmatic effort benefited from it. Six of the 8 (the 4-H 
studies exempted) examined how the programmatic effort was implemented and all 8 
examined in what ways clientele participated. Stakeholder involvement was always 
present and sometimes substantially so (e.g. Volunteers & Nutrition Education). The 
methods used were highly varied in how they approached clientele (current, potential &/or 
former). A survey often by mail &/or phone was not uncommon at times followed up by 
more intensive survey or site visits. Usually sample clientele were obtained from a roster 
of participants, However, for the 4-H studies random digit dialing was used to obtain 
clientele &/or groups to compare them with. In the case of Contemporary Youth, 4-H 
families were more willing to be interviewed than others. The 4 implementation studies 
all focused on how a programmatic effort was being carried out. Except for Leadership 
Development, all of them focused on efforts that were relatively new - at least at the time 
of the study. Stakeholder involvement is present in each one but not as extensively as 
for impact studies, with the exception of Leadership Development. The methods are quite 
varied with 3 of the studies using site visit teams with on-site observations and interviews. 
Two use a survey - one by mail and the other by phone. The 6 system/policy studies 
are more varied in the topics that they address than are the other studies with somewhat 
more varied stakeholder involvement and study methods. The nature and extent of 
stakeholder involvement may depend more on the importance or controversiality of a topic 
than on the fact that it is a system/policy type of study. Of course the same may be true 
for the other studies too. The methods are also varied ranging from teams of observers 
through phone interviews and mail surveys to an extensive literature search and synthesis 
intended to result in a new conceptual formulation. Overall impact studies focused more 
on the "how did it go and with what effect?' type of question while implementation studies 
focused more on the "how is it going?' type of question. System/policy studies in contrast, 
focused more on the "what might be?" type of question. 



* The appropriate references for these studies are: Impact - (Mayeske, 1982; Ladewig & Thomas, 1987; 
Steele et al., 1989; Allen & Rajotte, 1987; Brown & Hale, 1988a, 1988b; Steele, 1989; Makus et al., 1989) - 
Implementation - (Mtehael, 1989; TASCON, 1993; Rockwell, 1991; West et al., 1993) - System/policy - 
(Barthollc et a.l, 1987; Lippke, et al., 1987; R. Smith, 1989; Mayeske, 1990; Bottum, 1993; Bennett, 1992, 
1990). 



16-11 

313 



16.2 Tailoring Questions and Methods to Study Types 

These observations suggest that there are two general questions that can be used as 
"a lead in" to any programmatic effortAopic: (1) what is its nature and extent; and, (2) 
what benefits accrue (or might accrue) because of it? By breaking down these seemingly 
simple questions into a series of subordinate ones - they can be "tailored" to fit a 
particular circumstance. However, obtaining answers for them may entail a great deal of 
complexity. One breakdown of these questions is given in Table 16.4. We shall see in the 
sections that follow that the questions apply differentially for the different study types. 
They are most appropriate for impact and implementation studies and less appropriate 
for system/ policy studies due in part to the diverse nature of the latter. 

Let us examine what each question might entail. 

1. Where Is It being carried out? 

[This question has great salience for the impact and implementation type of studies but 
may not be as important for the system/policy type of study - sometimes because its 
answer is already known.] 

The question implies that we can specify a "universe" of units ( e.g. county office, county 
staff or other) that can be queried concerning the presence or absence of the effort, in 
Extension for example, the universe might be all States and territories or all counties. 
Similarly, within a single State it might be all counties. If a universe file is available then 
it must be up-to-date or be updated. For example, a universe file of staff must be updated 
periodically since leavers and new hires occur frequently, in the absence of a universe 
file one would have to be developed. If a system of records is available that contains 
descriptive information on these units (e.g. applications for project funding or payroll 
information for staff ) and it is judged that they do not suffer from any significant biases 
then they might serve as a universe file. If one has to be developed then it requires that 
a census type survey be conducted of all units gathering a very limited amount of 
descriptive information about each one. This descriptive information will usually be of a 
"non-judgmental" type (viz. not subject to much error) such as numbers of employees, 
numbers of buildings, geographic size, etc. thereby avoiding building errors into what may 
become the basis for selection of a sample. 
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Table 16.4 Generic Questions for an ln«Depth Evaluation of a Programmatic Topic 

1. Where Is tt being carried out? 

What is its geographic (or other) distribution? 

How many locales of the total are Involved? 
What is the incidence or concentration of effort by geographic locale? 
How "appropriate" are these locales compared to the others? 

2. Who participates? 

How many are there? 

What are their "key" attributes? 

How appropriate is it for them to participate according to different criteria? 

Are there non-participants for whom participation would be "appropriate"? 
What are their "key" attributes? 

3. In what ways do they participate? 

How often? 
For how long? 

In what kinds of "experiential environments"? 

4. What benefits result from the programmatic effort? 

How are participants different? 

Do some participants benefit more than others? 

Are there special considerations that may have an effect on how they 
benefit (e.g. unmet needs, unplanned for needs being met)? 

Are some practices or approaches more beneficial than others? 

Are there "exemplary " practices or approaches? If so, in what ways do they 
differ from the others? 

5. Are there other effects (positive or negative)? If so, what are their implications? 



Given the availability of a universe one can work with it in its entirety ( e.g. all 50 States) 
or plan to draw a sample from that universe using well known sampling procedures 
(Dillman, 1978; Henry, 1990). In Extension it is not unusual to use stratified sampling 
procedures to insure that the sample is diverse with respect to such concerns as 
geographic and rural-urban differences. For example, in the Volunteer study a stratified 
probability sample of 315 counties was drawn that ensured the inclusion of such 
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differences. In the Nutrition Education and Youth-at-Risk studies small random samples 
were drawn so as to ensure the presence of rural-urban, geographic and program focus 
differences ( the latter for Youth only ). For Leadership Development a sample of 3100 
staff were surveyed , stratified for State/territory and program area representation. For 
Water Quality Demonstration Projects there were only 8 sites available at that time and 
all were included in the study • a universe sample. 

The sample units may be asked to respond to a form (either mail or phone) which 
describes in some considerable detail the nature of their involvement with the 
programmatic effort. It is assumed that there is at least one person at each unit who can 
give a knowledgeable response. However, depending upon the topic the responses of 
several persons may be required. Very high response rates are attainable in surveys 
where the respondents are your employees or are recipients of funds from your 
organization. Rates of 100% should be planned for and budgeted accordingly so that a 
number of follow-ups to the units can be made. Additional time may also be required to 
do the follow-ups. A form of friendly persuasion coupled with a clogged persistence 
usually pays off even though additional time is required. However, the effort put forth 
more than compensates for an unbiased sample. In the study of Leadership Development 
an 86% response rate was attained from staff while an 87% response rate was attained 
for stakeholder interviews ( including Extension staff ) in the Nationwide System study.l'n 
the study of Volunteers in Extension a 99.6 % response rate was attained from the staff 
of a sample of 315 counties (Steele, et al, 1989). It would have been 100% save for the 
fact that the agent in one county had died recently and the Extension Director assumed 
that there was no one to provide the information. Actually there was a knowledgeable 
staff secretary who could have done so and had the Director permitted, the response rate 
would have been 100%. 

The information obtained in the survey can pertain to answers to some of the other 
questions in Table 16.4 as well. Some of these responses may entail a "good bit of 
judgment" on the part of the unit's staff. They may be asked to make judgments using 
definitions that are routinely used by them and therefore are quite familiar. At the other 
extreme they may be given definitions that have seldom if ever, been used by anyone 
and which therefore require a good bit of forethought and, if there is more than one 
person in the unit responding, perhaps a good bit of discussion between them as well. 
For example, one can easily count the number of participants in a nutrition education 
program if the program is structured so that an aide works with the homemaker in the 
home for an hour and one-half session once a month. The aide keeps a log of all this 
and the counting is easy. Imagine however, that we ask unit staff to tell us how many 
volunteers they work with in a year when the definition is anyone who "assists Extension 
or others through time, effort, funds or materials; is not paid by Extension; can be of any 
age and assist for short ( 2 hours or less) or long periods of time" (Steele, et al., 1989) 
Suppose also that many of the people who fit this definition are people the staff does not 
think of in terms of "their being a volunteer even though they clearly are (e.g. that's just 
Don the local farmer with whom I am doing the aquaculture demonstration project). As 
a general rule, the less familiar unit staff are with the definitions the greater the 
amount of attention that must be given to both pre-testlng and adequate but clear 
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and concise (as possible) instructions concerning their use. Pre-testing with a smati 
sample of unit staff must be done to insure that the definitions can be used in the way 
desired and the instructions must be developed as part of this pre-test to try to insure 
that the definitions will be used in the desired way. Perhaps more than one set of pre- 
tests and revisions may be required. 

The remaining subquestions are concerned with the concentration of effort by locale and 
the "appropriateness" of these concentratibns. Information on concentration by locale may 
come from the unit sample survey (or from existing files if dependable, or from some 
combination of both) while that for "appropriateness" can come from a couple of sources 
if the question itself is "appropriate". For example, in some programs we would expect 
there to be a concentration of effort where the "need" is greatest. However, for some 
topics like Volunteerism, the concept of "need" is not directly relevant. One might find that 
there are proportionately greater numbers of volunteers in sparsely populated than in 
densely populated areas and this might be a reflection of the nature of the communities 
as much as it would be anything else - it does not necessarily reflect the results of 
a decision concerning the allocation of resources. However, when such a concern 
is appropriate, external sources of information might be brought in to perform comparative 
analyses of how the allocation of resources relates to these external criteria For example, 
in the Study of Nutrition Education for Low-Income Families, data on the resources 
allocated to each site were analyzed against such external criteria as the number of low 
income families residing in the area, the number of families participating in different 
support programs (e.g. Food Stamps; Women, Infant & Children Program, etc.) to 
examine the extent to which the concentration of resources was in alignment with these 
external data (Mayeske.1982). 

2. Who participates? 

[ As with the prior question, this one may have slight salience for system/policy studies 
but is of great importance for impact and implementation studies. If it is of importance for 
the former, answers can perhaps be obtained by recourse to extant information.] 

How might we approach current or potential clientele of a program? We have four 
options: (1) mail; (2) phone; (3) face-to-face (including structured group interviews such 
as focus groups); and, (4) some combination of the preceding. One cannot meaningfully 
select among these until decisions have been made concerning the level of credibility 
needed for the information that is collected and the cost associated with attaining that 
level of credibility. For example, perceptual information can be obtained readily by all of 
these. But, if right and wrong types of information are to be obtained or specific 
competencies are to be demonstrated then a face-to-face setting may be required to 
ensure that the respondent actually has the competency. Alternatively, if credibility is not 
as acute then this same information might be obtained via the phone or even a mail 
survey. Or, if the information sought is such that an individual may not have a sense of 
it that can be articulated or highly critical information is desired, then some form of focus 
group setting might be most appropriate. 
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To fully address the issues of who is receiving or might be in "need" of assistance, non- 
target as well as target areas need to be surveyed. That is to say, one must address 
not only how many actual and potential participants there are in target areas but also in 
areas where the programmatic effort is not currently present as well. Because this kind 
of coverage can substantially increase study costs, PE's or their clients may opt for target 
area information only. In so doing they may be giving up not only their ability to 
adequately address the issue of "need" but also give up a potent source for purposes of 
obtaining comparison groups as well. The major concern is how to get to the universe of 
possible participants (current, former and potential). There are two ways in which such 
a universe might be accessed: (1) by sampling from a roster ( that already exists or has 
been especially generated for purposes of the study); and (2) random digit dialing. Let's 
consider the latter first. 

o Sampling by Random Digit Dialing - in this procedure telephone numbers are 
generated randomly, the numbers called and the respondents "screened" as to whether 
or hot they are now or ever have been involved with the programmatic effort. In that initial 
contact or in some follow-up call, mail-out or personal visit, information on the 
respondents status with respect to "key" attributes is obtained. "Key" attributes are ones 
that are related to the kinds of targeting efforts outlined in previous chapters - 
individual/household/community/business enterprise attributes related to "need" for the 
programmatic effort as well as other socio-demographic factors. Information is also 
gathered on the nature and extent of their participation and how they might have 
benefited from participation ( questions 3 and 4 in Table 16.4). 

This approach has certain obvious advantages and disadvantages. Its main advantage 
is that it avoids dependence on rostertype information which can be loaded with unknown 
or unspecifiable errors, it also avoids any knowledge on the part of program staff as to 
who might be a respondent as well as any behavior that might result from such 
knowledge (e.g. forewarning, coaching, biasing responses, etc.).lt also has many 
disadvantages. It requires the use of a telephone which may not be appropriate for some 
audiences - low-income households may not have one. Another serious concern is that 
the respondent may not be the appropriate one to be interviewed - it could be a non- 
residential number, a FAX machine or a child home alone. An even more serious concern 
is that the respondent is the one wanted for interview purposes but refuses to be 
interviewed. This is an increasingly common concern and contributes to substantial non- 
response error. This error is compounded by the fact that those who have had some 
contact with the programmatic effort are more inclined to be interviewed than those who 
have not,as seen in the 4-H studies. This approach could be extremely expensive and 
not very productive if the incidence of "need" or the programmatic effort is small relative 
to the total population - far too many calls would have to be made to find out anything. 
[ See Dillman (1978) for more on the pro's and con's of this method.] In such a case 
some other method such as use of a special roster would be preferable. 

o Sampling From a Roster - in the unlikely event that a roster is available which 
allows one to identify a universe that is inclusive of the target population, the PE's task 
is greatly simplified. For example, in an instructional setting student records may be 
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dependable enough to use. Or, information may be available in a county which is current, 
relevant and dependable. Or, commodity associations may have up-to-date membership 
lists, etc. Often however, rosters that are available are rife with error that is well nigh 
impossible to get rid of. Consider for example, a list of all persons who have been issued 
a driver's license in the past three years in State Z as a potential source for an 
evaluation of a seat belt safety program. Since residential changes are frequent and the 
factors related to residential changes are many (e.g. age, employment status, educational 
level, etc.) the roster is biased in ways that are difficult if not impossible to correct - those 
who moved have no known address; hence, are unreachable. Voter registration rolls and 
telephone directories have similar problems. 

The development of a special roster may require considerable time and resources. The 
only way to provide the information needed may be to put actual enumerators in the field 
to conduct a complete census.However, the cost may be prohibitive. Clearly, there is a 
trade-off in terms of the accuracy of the information needed relative to the cost of 
obtaining it. For some purposes one might be able to tolerate less accurate information 
(e.g. when the lack of time precludes getting more accurate information). 

Let us assume that however produced, we now have a universe roster that includes 
clientele (current, former, prospective, never). [Needless to say, the more information that 
is available on them the more a sample can be stratified prior to contact.] We shall 
assume that the sample drawn from the roster will have to be contacted via a letter or 
post-card to let them know that a contact will be forthcoming. [ In the case of a mail 
survey, a direct mail-out may be possible.] This follow-up contact will probably be by 
phone to obtain the desired information or to schedule the person for a face-to-face 
setting, either by coming to where they are or by their coming to a central location. For 
this latter contact the same kinds of "key" attribute, participation and benefit information 
are obtained as described in random digit dialing. 

For phone interviews, 20 minutes is about the maximum one can hope to get of a 
person's time. For mail surveys - the shorter and simpler the form the better - even 20 
minutes of a person's time may not be attainable. For focus groups, substantial attrition 
can be anticipated in the numbers contacted versus those who actually show up for the 
focus group meetings. Exceptions are for persons who have some close connection with 
the programmatic effort. 

What to do 'about those who refuse to be interviewed or respond is a vexing problem. 
Ideally one could just replace them with another random draw. However, those who 
refuse are likely to be systematically different from those who don't in ways that can affect 
the survey results. It is desirable to have some information on the non-respondents so 
as to gauge the ways in which their absence might have affected the results. This is of 
course easier said than done and is a task to which one can apply great ingenuity. For 
example, one might follow-up on a random sample of the "refusers" with an offer to 
reward them in some way ( e.g. money, prize, etc.) to provide certain socio-demographic 
information which will enable comparisons to be made with the respondents. 
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What kinds of response rates should be planned for? Ideally one should try to attain 
100% but with the general public such a figure is not likely attainable. Short of 100% 
however, the concern is with the biasing effect of non-response on the results 
rather than an absolute percentage of response (viz. 95% is not "good" if the 5% non- 
response were extremely different). If time and resources permit some trial tests might 
be conducted to better understand the nature of non-response. For example, a face-to- 
face interview with some non-respondents to a mail or phone survey might yield insights 
into the nature of non-response. Indeed, some such interviews might be a vehicle in 
themselves for raising the response rates. Dillman (1978) cites an example where non- 
respondents to a mail survey were followed up with a face-to-face interview. This served 
to increase *he response rate substantially and showed that the reason for non-response 
was a lack ot sufficient literacy to provide a written response to a mail survey. As a last 
resort one can perform "worst-case scenario" kinds of analyses to tease out what effects 
non-response might have had on the results (e.g. how different on a number of factors 
would the non-respondents have to be to distort the results and how likely is it that thev 
are that different). 

All of these concerns are important to obtaining estimates for the number of participants 
and non-participants.The size of the sample needed to obtain these estimates will 
depend on cost as well as on the level of precision desired. On occasion some breadth 
of coverage is sacrificed for greater depth of coverage. 

3. In what ways do they participate? 

[ As with the prior questions, answers to this question may be obtained in a very different 
manner for system/policy studies than for the others. Further, for implementation studies 
the emphasis may be on the appropriateness of what is provided or the difficulties in 
doing so, rather than on their possible results.] 

Both current and former participants will have had experiences with the programmatic 
effort that they can reflect upon. They can do this using a completely open ended format 
or one that is highly structured and perhaps highly detailed as well. An open ended format 
might use very general questions to which the respondent could "plug-in" their relevant 
reflections. For example, one could ask what their overall impressions are, what they liked 
most about the program, how they felt they benefitted, what they felt could be improved 
etc.The problem with an open ended format is that if much time has elapsed since their 
participation they may have forgotten many details and hence give somewhat vague 
responses. A more structured approach might entail the development of an experiential 
descriptor checklist that would enable the interviewer to check off aspects of the 
respondent's educational experiences as they are described by them ( or to them, as 
appropnate). If appropriate developmental work was done forthe Educational Experience 
Profile in the program development cycle, then it should not be difficult to develop such 
a descnptor for a single program or even a family of programs. If such profile information 
is not available or the programmatic effort is very diverse, then some considerable 
developmental work may be required as part of the in-depth evaluation. The concerns of 
frequency ( how often), recency (how long ago ), intensity (group, individual, etc.) and 
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duration ( for how long) would be covered in such a descriptor. 
4. What benefits result from the programmatic effort? 

[ Implementation studies do not usually deal with this question except on occasion to 
estimate probable impact. However, impact and system/policy studies do but in a very 
different manner. System/policy studies usually focus on the possible effects of some 
change but their means of doing so are highly varied and difficult to describe. For impact 
studies, something like the following may be appropriate.] 

In a similar vein one can obtain information on benefits in an open ended format or one 
that is highly structured. It is conceivable that one could have a highly structured ,. 
experiential descriptor with an open-ended format for benefits so as to avoid suggesting 
benefits to the respondent. For example, "how do you feel that you benefited from your 
participation in program XYZ?" Smith (1991) has developed a moderately structured 
format which emphasizes the concepts of relevance, quality and utility. This format is 
reproduced in Table 1 6.5. Even if not used exactly as presented by Smith, these concepts 
can serve as useful guides for the development of an interview format. 

Just when would it be appropriate to have a series of structured questions about 
benefits?. There may be times when the need to document the occurrence or non- 
occurrence of a benefit overrides the concerns about leading the respondent. Or, there 
may be such a complex array of them that some will be overlooked if not mentioned 
explicitly. For such situations a benefit checklist might be developed which allows the 
interviewer or respondent to check off the benefit as it is cited ( or read to the respondent, 
as appropriate). If there are special considerations that affect benefits they need to be 
examined as part of the benefits questions. Again these can vary from open-ended 
questions such as "are there other things that could have helped you benefit more?" to 
"which of the following best describes your situation as you participated in program 
XYZ?". Or, for unplanned needs being met "are there other ways in which you feel that 
you benefited from your participation in program XYZ?" to "are any of the following other 
ways in which you benefited from your participation in program XYZ?". 

For some programmatic efforts the.notion of how persons like the participants would have 
fared without the program is not an important one because it is obvious that the 
participant couldn't have acquired what they did elsewhere or without the program - the 
benefit is self-evident. Or what they could or could not acquire in the absence of the 
program is not of sufficient priority or relevance to be of concern. For example, how much 
or little children learn in the absence of schooling ( or instruction) is not a question of 
interest. However, how to make schooling better is a question of interest and much time 
and energy is focused on identifying practices that can be used to improve schooling. 
However, for other programmatic efforts the question of why should one have the 
program at all is an important and relevant one that bears on the allocation and utilization 
of scarce resources. For them the question takes on considerable significance, We can 
recast the discussion by talking about the effectiveness of one practice versus another 
or of one approach ( a package of practices) versus another. One kind of approach is 
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Table 16-5 Generic Evaluation Format Emphasizing Relevance, Quality 
and Usefulness 

Program Identification 



Please rate this program on its relevance, its quality and its usefulness to you. To answer circle a number at the 
end of each question. Space is provided for comments. Thank you very much. 

(Cirde number) 

RELEVANCE Yes, very No. not 

much so at all 

1. Did this program deal with important needs or 
concerns for 

a. you or your family? 5 4 3 2 1 

b. others in your community? 5 4 3 2 1 

2. Was the content what the program announcements 

said it would be? 5 4 3 2 1 

Comments about the relevance of the program: 



QUALITY 



Yes, very 
much so 



No. not 
at ail 



3. Was this program 

a. current up-to-date? 

b. understandable? 

c. presented in an interesting way? .... 

4. Did this program meet its advertised objectives? , 



5 4 

5 4 

5 4 

5 4 



3 
3 
3 
3 



2 1 

2 1 

2 1 

2 1 



Comments about the quality of the program: m 



USEFULNESS Yes, very No. not 

much so at all 

5. Did the program answer any questions you had? 5 4 3 2 1 

6. Did it help you learn how to do new things or how 

to do others better than you knew before? 5 4 3 2 1 

7. Do you intend to use or have you already used any 
of the information or skills that you acquired. 

from this program? 5 4 3 2 1 

Comments about the usefulness of the program: 



(CIrde number) 

8. OVERALL, was this program worth your time to attend? 

Yes, very No. not 

much so at all 

5 4 3 2 1 

Adapted from Smith (1992) Criteria for Excellence. 

College Park, MD: University of Maryland Cooperative Extension Service 
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none at all, namely the absence of a program - a kind of zero point . How best to develop 
this zero point has been much discussed among social scientists and has generated a 
considerable body of literature (Shadish, et al., 1991). 

Suppose that we think of these matters in terms of our having a yardstick by which we 
can record the movement that a group makes from one point in time to another, say from 
point x, at time 1 to point y, at time 2. Assuming that they are not just random 
fluctuations, we can compare the magnitude of movement for one group versus another 
to help us decide just how useful or valuable their approach ( or practices) might be. If 
the group that represents the zero point (the no program group) is of sufficient importance 
to be included then the comparisons can be particularly instructive. The most widely 
recognized way of producing a zero point is through random assignment of potential 
participants to experimental ( those receiving the program or approach) or control (those 
not receiving any program or approach but may receive some kind of placebo) groups. 
[If the zero point is not of intc rest, then random assignments are made to different kinds 
of experiential groups - one of which may be "the old way of doing business".] If 
achievable and sustainable, such randomized assignments are considered to be the 
best way to produce such points on the yardstick for purposes of comparison with other 
groups. However, there are many other ways to obtain values for purposes of 
comparison - a comprehensive list of these is given in Rossi & Freeman, (1993 ,see esp. 
pp's 243-259). They all suffer from various shortcomings when compared with the random 
assignment approach. Mood (1983) reminds us that even when random assignments can 
be attained, experiments can produce effects which may render them inapplicable to the 
whole population. Among these are: the volunteer effect (those who agree to participate 
in the experiment are committed to it and hence perform better than otherwise); the 
Hawthorne effect (those selected to participate have an enhanced sense of importance 
and hence behave differently than otherwise); size effects ( small groups &/or small 
programs are easier to work with than are large ones and experiments tend to be of the 
former); administration effects ( sponsors of an experiment tend to be highly 
conscientious about insuring the success of the effort but on a large scale the 
competencies and commitments of the staff may be far more varied and less 
enthusiastic). [These effects also apply to "pilot" & "innovative" efforts.] 

Exemplary practices or approaches ( a package of practices) are often identified through 
a subjective judgmental process that looks for extremes - practices or approaches that 
are "unusual" in one or more respects. For example, the practices can be considered 
unusually effective or efficient, or innovative, or very different in the way something is 
accomplished or produced, or they epitomize the very "best" of one's programming 
efforts, etc. It is often desirable to include in an in-depth study a non-statistical sample 
of exemplary practices so as to insure that some extremes are included that are 
sufficiently infrequent and hence might not appear in a statistical sample. By collecting 
the same kinds of information on them as on the statistical sample, one can make a 
variety of comparisons which could prove to be very instructive for future programming 
and for program improvement. 
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One might also conduct a series of case studies of some approaches that appear to be 
"meritorious" in some respects as identified from the analyses of the statistical and 
exemplary samples (Yin, 1989). For example, some could be unusually effective or 
ineffective and a team of observers might go into the field to document and/or gain a 
greater understanding of what is happening by observing the approach in action, by 
talking with local staff and clientele.etc. 

The use of cost information can be most revealing - especially if one has satisfactory 
comparison groups - but it is usually overlooked (Scriven, 1991). Cost analysis and cost- 
effectiveness analyses whether using budgetary information or by converting resources 
used to cost figures can give yet another valuable perspective to in-depth studies (Levin, 



5. Are there other effects (positive or negative)? If so, what are their implications? 

[ This question is applicable to all three types of studies.] 

Basically this question is concerned with spin-offs that are produced by the programmatic 
effort - unplanned occurrences - which can be anticipated, like those in Table 9.33 , 
or unanticipated. Those that can be anticipated are usually positive in nature. It is the 
unanticipated one's that are likely to be negative and require detection by an observer 
who has keen powers of observation and a healthy degree of skepticism in orcler to pick 
up such things as the program making the problem worse or creating another problem 
that is similar in magnitude or even worse than the one it is attempting to ameliorate [see 
Sieber (1 981 ) on "Fatal Remedies"]. Consider for example, a technology that dramatically 
reduces soil erosion while simultaneously dramatically polluting ground and surface water. 
Or, a program that creates a dependency of the clientele on the staff. Or, a program that 
decreases teen pregnancies but dramatically increases teen sexual activities. Or, the 
program works but the clientele can't stand it ( sometimes called the "listerine" effect). 
The list of examples could go on but It suffices to say that a PE must be aware of such 
possibilities plus the occurrence of flat out fraudulence and incompetence and must be 
in a position to point them out if they are observed. 

16.3 Some Issues in Data Analysis 

Up to this point we have said very little about the analysis of data from in-depth studies. 
For Impact and Implementation studies there are any number of books that provide 
good guidance concerning analysis (e.g. Rossi & Freeman, 1993) whereas for 
system/policy kinds of studies the range of possibilities is very large entailing anything 
from "pondering with pen in hand" to sophisticated types of analyses entailing both 
qualitative and quantitative methods ( see especially Patton,1990 for the former). Ideally 
the analyses would flow from the study questions aided by a number of detailed steps. 
For many such studies there are at least seven concerns that often arise and can be 
particularly vexing: (1) what to do about missing data; (2) how should items be scaled; 
(3) how to handle weighted data; (4) how to handle mixed levels in analysis; (5) whether 
or to what extent data reduction techniques should be used; (6) what models are 
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appropriate for data analysis purposes; and, (7) what kind of a difference is worthwhile? 
Let us consider each in turn: 

1. What should be done about missing data? 

Suppose that some of the respondents did not respond to ail of the questions or items 
or practice exercises. How can they then be included in the analysis along with those for 
whom a complete set of responses are available? A common practice is to assign them 
the average (or mean) of those who did respond on the assumption that they would be 
somewhere near the average if they had responded and recognizing that such an 
assignment would not greatly distort the values for the others. However, if they are not 
near average then such an assignment could be misleading not only for the analysis of 
that particular item but for it's relationship with other items or variables as well. If there 
is a single variable (or composite of variables) that is of overriding interest then each of 
the items can be analyzed against this "criterion" variable with a category especially 
created for the non-respondents. For example, suppose the "criterion" variable is the total 
score on an achievement test and that we are using it to determine what value we should 
assign to students who did not respond to "how many hours a week do you spend 
watching television", as follows: 



Hours Per Week Spent Watching TV 


Average Achievement Test Score 


31 or more 


Low 


21 -30 


Medium 


11-20 


High 


10 or less 


Low 


Non - respondents 


Medium 



Such an analysis would help us to decide how to classify the non-respondents. In this 
hypothetical case they would be assigned a scale value similar to those in the 21 - 30 
hours category. The same procedure could be used for other items where missing data 
is a problem. Such a technique has been used extensively in studies of educational 
programs ( Mayeske et al., 1972; 1973a; 1973b; 1975). When a criterion variable is not 
available an algorithm cairbOTsedto-estimate values for missing data which assigns 
values in an iterative manner. Suffice it to say that if there is an extensive amount of 
missing data any imputation and analysis based thereon can be extremely misleading. 

2. How should items be scaled? 

Many items have a natural ordering. For example, actual height or weight can be used 
in their everyday metric for analysis purpose. For others, a natural ordering is not 
apparent. For example, an item with a "yes" or "no" response does not have a natural 
order yet it is easy to assign a higher value to the "yes" than to the "no" (e.g. yes=2, 
no=1 ) and thus the item is easily entered into analysis. However, other items are not so 
readily ordered. Consider for example, the creation of a variable designating the region 
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of the country in which one resides (e.g. Northeast, North Central, West, South). If one 
wants to create a single variable for analysis purposes it is not apparent how this should 
be done. One way is to create some arbitrary ordering ( e.g. Northeast » 1 , North Central 
= 2, West = 3, South ■ 4). However it is difficult to ascribe meaning to such a variable let 
alone interpret its relationship with other variables. As in the previous example here 
again if a "criterion" variable is available an analysis like the preceding can be conducted 
to guide in the assignment of scale values. When the averages for the "criterion" variable 
are used as the scale values then the linear relationship of the item with the "criterion" 
is maximized (Beaton, 1969; Mayeske, 1969; Mayeske et al., 1972). In the previous 
example, the use of such values would have "linearized" a curvilinear relationship and the 
meaning of the item would have changed from "# of hours spent per week watching TV" 
to something like "optimum # of hours watching TV per week, as related to achievement". 
Such a technique can be extremely useful when one has a large number of items with 
categories that do not have a natural ordering. Then too, the use of such an analysis 
readily reveals curvilinear relationships. If such a criterion is not available, then a variable 
or set of variables that are of especial interest might be used to guide the assignment of 
scale values. 

3. How should weighted data be handled? 

When stratified sampling procedures are used one might be lucky enough to have a 
sample that is self-weighting ( viz. no special weights are needed because the strata 
contribute to the sample in proportion to the population). More often however, some set 
of weights must be applied to the data to correct for disproportionate sampling. Such 
weights must be built into the analyses unless it is found that they make no appreciable 
difference in the analytic results. One can only be sure of this by doing both weighted and 
unweighted analyses. If a number of higher order analyses are done it can be 
exceedingly burdensome to do two sets of analyses. In studies that involve multiple levels 
the situation may become even more complex with each level having its own set of 
weights. For example, in a study of educational opportunities there were separate sets 
of weights for schools, for teachers and for students ( Coleman, et al., 1966; Mayeske et 
al., 1972, 1973a, 1973b, 1975). 

4. How should mixed levels, be handled for analysis purposes ? 

It is not uncommon to find that different levels are entered into the same analysis 
framework. In the previous example, the levels were: students; teachers; schools; region 
of residence; and, rural-urban location. For these levels there is an upper limit to their 
explanatory power which can and should be computed so that the analyst can know what 
proportion of the total variation that could be explained at that level is actually explained. 
Consider for example the earlier example of student achievement. Suppose we observe 
in a mixed levels analysis that 3% of the total variation among students in their 
achievement is associated with their region of residence.lf we also know from an 
among/within levels analysis that 3% is the maximum that could be explained by regional 
differences, then we would not look further into region as an explanatory variable. We 
might however, perform analyses of the regional differences with other variables at that 
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level to better understand the possible origin of these differences. For some pulses 
such analyses at different levels may be preferable to lumping them all together in a 
mixed format. These analyses are called by various names such as levels analyses, 
among/within analyses or more appropriately as multivariate analysis of 
variance/covariance (see Mayeske & Beaton ,1 975 for extensive use of these techniques 
as well as means for determining the maximums at each level). 

5. Should data reduction techniques be used? 

With any large data base it is often difficult to "see the forest for the trees". Data 
reduction techniques can and should, be used to "boil the data down" into a number of 
indexes or composite variables that are meaningful yet vastly fewer in number than single 
items or even single variables. For example, items related to a family's occupation, 
income, property and valuables owned, etc. can be combined into an index of socio- 
economic status or of general affluence, provided of course that the items are sufficiently 
interrelated to support their being combined. Factor analytic techniques can be used in 
a variety of ways to guide in making these reductions (Mayeske et al., 1972, 1973a, 
1973b, 1975). Similarly, variables can be combined into higher order composites which 
can then be included in a number of higher order analyses and models ( Mayeske & 
Beaton, 1975). 

6. What are appropriate models for data analysis purposes? 

Ideally there will be a logic model for data analysis purposes which specifies the 
hypothesized relationships between the program variables, intervening or intermediate 
variables and outcome variables. This logic model will not likely be the same as the 
program logic model but will draw upon many elements from it ( if there is one available). 
Again, multivariate analysis of variance/covariance will likely be the analysis model of 
choice perhaps combined with path analysis or some form of structural equation modeling 
(Mayeske & Beaton, 1975). Outliers may be examined for the existence of curvilinearities 
and discontinuities with exploratory analyses being done to try to understand them 
(Mayeske & Beaton, 1975). Partitioning of the variation in a dependent variable or set of 
variables may also be used to avoid undue reliance on the interpretation of regression 
coefficients which may, under certain conditions, be extremely misleading ( Mayeske & 
Beaton, 1975). Or, a form of weighted regression might be used to approximate 
longitudinal growth curves using cross-sectional data (Beaton, 1986; 1984; 1983; 1982; 
1981; 1975; Beaton & Mayeske, 1986a, 1986b; Mayeske, 1982; Mayeske, et al., 1972) 
In short, an eclectic use of data analysis models is advised rather than relying solely on 
one. 

7. What kind of a difference is worthwhile? 

This question takes different forms depending upon the type of study involved and the 
kind of analysis models being used. Basically it is concerned with the values that can be 
placed on a difference or improvement attributable to a particular program. The 
difference or improvement may involve pollution, destructive behaviors, cognitive or 
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physical proficiencies, birth weights, agricultural production, business practices or a wide 
variety of other phenomena. Assuming that these differences are not due to random 
fluctuations, the concern is whether they are worthwhile in some other sense as well. 
Cost can be one kind of information brought to bear on such considerations and if cost 
data is available it can be usefully entered into the analyses. Even if cost data is available 
however, some other values and concerns must come into play. Judgments have to be 
made as to whether the difference is of sufficient magnitude to be of value in it's own right 
( if it is a consequence or impact) or of a sufficient magnitude to lead to some desirable 
end. For example, if pollution is reduced, is the reduction large enough to help us reach 
a desired level in some given time period at an affordable cost? Or, is the enhancement 
of the cognitive proficiencies of youth sufficient enough to hold up over a sufficient period 
of time so that they can benefit in the longer run from later experiences that build upon 
these skills? Or, are the enhanced business practices sufficient in magnitude and type 
to enhance the viability of the business and the community of which it is part? In short, 
value judgments inevitably come into play in deciding the worth of a particular result. 

It may be of interest to see how these questions might be dealt with for system/policy 
type studies, especially those that involve the analysis of qualitative kinds of data. The 
order and phrasing of the questions will change somewhat in the discussion that follows: 

1. Should data reduction techniques be used? 

Most qualitative studies ( including those that analyze the content of field notes, 
documents, recordings, verbal exchanges, etc.) need to "boil down" their data in some 
form so as to better manage and interpret it. Often such analyses will develop some 
system of categories into which segments of the narrative data can be categorized. An 
initial set of categories is sometimes developed by a careful review of the narrative 
statements which are then successively refined as segments of the narratives are 
classified into them. These segments may be developed by a technique akin to "parsing" 
in which narrative statements are broken down into component parts that exemplify the 
hypothesized concepts/categories. Once all of the segments have been identified and 
classified, further analysis can then be performed on them - both quantitative and 
qualitative. The success of this effort depends upon the extent to which another analyst, 
working independently.could come up with the same segments, categories and 
classifications if given the decision rules for their identification and classification. If 
agreement can be reached at some acceptable level, then higher order analyses based 
upon them are justifiable. If agreement cannot be reached then the analyses may reveal 
nothing more than the idiosyncrasies of the analyst. 

2. What should be done about missing data? 

If there is missing data, by which is meant omissions in the narrative statements, some 
form of imputation may not be possible nor desirable. The fact that it is missing, if not due 
to oversight, may be an important source of information in its' own right. 
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3. How should the data be scaled? 

If the data are to be systematized and analyzed quantitatively the segments or categories 
of segments might be rated on a variety of scales for the extent to which they exemplify 
different attributes with these scale values then being analyzed with the segments as the 
unit of analysis. These ratings can be related to one another as well as to other attributes 
of the segment such as demographic descriptors, situational characteristics, nature of the 
interaction, etc. 

4. & 5. How should weighted data and mix* je handled? ' 

If a stratified sampling procedure is used tc obtain int idrratives then some kind of 
sampling weights may be needed to correct for disproportionalities.Usjally however, the 
sample is drawn in such a manner that it is either self-weighting or random sampling 
procedures are not used and therefore weighting is not an issue even though one's ability 
to generalize beyond the sample remains an issue. Unless, of course one is working with 
the entire universe. Levels analyses may also be possible with such data. However, the 
occurrence of such analyses is rare. 

6. What are appropriate data analysis models? 

To the extent that the analyses are intended to be quantitative many of the procedures 
cited earlier might be applicable. In addition, a variety of other quantitative techniques nay 
be appropriate, especially for content analyses (Sedgewick, 1983; Krippendorf, 1980). 
However many qualitative studies/analyses are conducted in order to understand or 
uncover phenomena that cannot be dealt with adequately using the usual quantitative 
approaches, Hence, the analyses too may retain a qualitative nature. 

7. What kind of a difference is worthwhile? 

If the analyses are quantitative in form then the same concerns expressed earlier may 
also apply here. However, if their qualitative nature has been preserved then they may 
take on a very different form.Conceivably, the qualitative effects of a program may enable 
the comparison of "apples, oranges and even green pears" as noted by Patton (1990). 
For example, what one person might derive from their participation in a program may be 
totally different to that if another person. Hence judgments concerning practical 
significance must still be made but they are not necessarily based on magnitudes. 

16.4 A Sequential Approach for an In-Depth Evaluation Study 

One can imagine a situation in which a Program Evaluator (PE) is called in to confer with 
a potential client concerning an in-depth evaluation study. The PE may bring to the setting 
a broad background in program evaluation and research methods. The client's 
background may range from absolutely no knowledge of evaluation to an advanced 
degree in a technical area or in human development with some graduate level work in 
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evaluation. The client's motivation for the study may vary widely as well - from a genuine 
desire to learn about their programmatic effort to little or no enthusiasm but having to do 
so due to some requirement. The backgrounds of both parties plus the motivation for the 
study and the nature of their relationship with the PE (e.g. known and trusted versus third 
party alien) will greatly affect the nature of their deliberations. One can view these 
deliberations as a kind of negotiating session (or series of sessions) in which the two 
parties (actually there may be more than one person in each party) exchange views as 
to what the questions are that the study will address and how they might be answered. 
There is thus an iterative relationship between the study questions and the study design 
- each one in turn delineating the other. Hence, the first step in a generic sequence of 
steps in an in-depth study can be depicted as two highly interrelated tasks called: 
Determine Study Questions; and, Develop Study Design, as in Figure 16.1, with the 
arrows to and from representing the negotiations and the longer box for Study Design 
representing more detailed specifications once the negotiations have been completed. 

There is a set of at least seven concerns that come into play in these negotiations that 
we shall call clarity, credibility, coverage, cost, accuracy, form(at) and time. As depicted 
in Figure 16.2 they also play back and forth on one another in a very rapid fashion. 
Indeed, the PE might be envisioned as a juggler who keeps all of these in motion at once 
but is also skilled enough to change their order while they are being juggled. Let us 
examine each of these seven in turn: 

oo Clarity - on the one hand the client may not be at all clear as to what questions 
they should be asking. Alternatively, the client may have a well developed notion of what 
the questions are and even how they might be addressed. In the first situation the PE will 
help the client clarify their thinking taking into account such concerns as the maturity of 
the program, the uses that will be made of the study results, the specificity of the 
programmatic goals (e.g. enhance self-esteem versus impart skills x,y &z), etc. In the 
second kind of situation the PE's task may be greatly simplified if the client has an 
adequate and appropriate set of questions and design in mind. However, if they are not 
adequate and appropriate then the PE may face the difficult task of tactfully reorientinq 
the client. a 

oo Credibility - will the credibility of the study be an important concern and how 
might credibility concerns be manifested? Will who does the study and their relationship 
to or independence of the program be a factor in influencing the acceptance of the 
results? How does the nature of participation or occurrence of benefits ( if such is an 
appropriate concern) need to be documented? Will the verbal report of the participants 
(or former participants) suffice or will more specific documentation be needed? Will 
observations of a third party observer suffice?, etc. 

oo Coverage - to what extent are breadth of coverage and depth of coverage of 
concern and to what extent can they be. traded off? For example, is it necessary to focus 
on the entire programmatic effort or can just some of its' components be examined? Can 
fewer units/sites be studied in more depth or must strong generalizations be made using 
many units/sites?, etc. 

oo Accuracy - this term is used to cover a variety of technical concerns such as: 
whether statistical generalizations are desired and if so, with what degree of precision; 
in what ways and how adequately can participant changes be gauged (e.g. the concerns 
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Figure 16.2 Considerations Affecting 
Study Purpose & Scope Negot i at ions 
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of reliability and validity of measurement), etc. 

oo Cost - the resources that can be made available to conduct the study is a 
potent factor which will affect many aspects of the questions addressed and the study 
design. Resources can be thought of as covering more than just money, For example, 
in-house staff might be used for some tasks or even to conduct the entire study thereby 
avoiding the need for extra funds, etc. 

oo Form(at) for Results - the form(at) in which results are needed can also be 
a factor in determining the speed of completion. Clearly, raw data or crudely tabulated 
data can be produced more readily than can an interpretive report and the latter more 
readily than a videotape of the results. 

oo Time - when the study results will be needed will play a large role in 
determining what can be done. If little time is available many of the other concerns 
become irrelevant. As noted earlier, in-depth studies may require many months to a few 
years to complete and if such time is not available then alternatives need to be 
generated. One alternative for a paucity of time is to consciously decide to not do a study 
recognizing that a proper one could not be done in the tjme available. 

The next steps in the generic sequence follow directly from the completion of these steps. 
[It is not uncommon for outside assistance to be sought at this time to carry out the next 
steps.] For an impact or implementation study there will undoubtedly be some concern 
with literature and background document review, sample selection, instrument 
development,test and revision, field data collection plans and procedures, analysis plans 
and procedures, synthesis of results and report preparation, etc., as depicted in Figure 
16.1. For system/policy studies however the content of the steps is less, clear. In the 
examples examined earlier two of them did involve some of these concerns while the 
other entailed a large scale search, review and synthesis of extant literature, study 
results, background documents, minutes of meetings, etc. in order to develop a new 
conceptual model. It is as if the nature and magnitude of the steps change depending 
upon the idiosyncrasies of the particular system/policy study. 

Stakeholder involvement can occur at three different points in a study. Their involvement 
can be sought to help define the study questions, or to advise on the relevance and 
appropriateness of different procedures and instruments or, to aid in critiqueing the results 
and in determining their implications. Stakeholder involvement is portrayed in Figure 1 6.1 
as occurring at these different points in the tasks. [ They can also be involved in 
reviewing progress but their influence there is relatively minimal.] Their involvement in the 
first two tasks (viz. 1a and 1b) could increase the amount of time required to complete 
them. This might be ameliorated somewhat if they are included in the initial discussions 
even though it is not the purview of the PE to do so at that point in time. However, the 
PE might suggest to the client that they might want to have some stakeholder 
representation at this meeting, if appropriate. 

In the next chapter we examine ways of sharing and using results whether they come 
from an in-depth study or from a Life-Cycle Program Evaluation. 
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Chapter 17.0 Sharing and Using the Results of Life Cycle Program Evaluations 
and In-Depth Evaluation Studies 

In this chapter we review a variety of different uses that can be made of evaluations and 
then focus in detail on those that are planned uses of a study's results. Examples are 
drawn from a number of studies conducted by the Cooperative Extension System in 
recent years. 

17.1 Varieties of Utilization 

We can identify a variety of ways in which evaluation studies can be used. The most 
familiar kind of utilization and the kind we will devote most attention to is that of the 
planned or intended use of study results. However, there are also fortuitous uses 
and uses that occur independently of a study's results, usually while the study is in 
some stage of implementation. We shall call this latter non-results related uses.We may 
create a two-way table for these, as follows: 



Table 17.1 Varieties of Utilization 








FORTUITOUS 


INTENDED 


NON-RESULTS 


a 


b 


BASED 






RESULTS 


c 


d 


BASED 







In cell a we have fortuitous or unanticipated use of a study that is not based upon its 
results. This may occur in a variety of ways. One example is to use the existence of a 
study to postpone any changes being made (usually of a resource reduction nature) until 
the study results are in. Another, is to postpone any other studies or audits being made 
of the program until the current one is finished due to the excessive burden that would 
be placed on staff and possible. confusion due to a number of different activities going on 
simultaneously. In cell b we have intended uses that are non-result based.These can 
occur when, as part: of a strategic or tactical objective ( as discussed in the prior chapter) 
one deliberately starts an in-depth study so as to be able to "fend" off actual or potential 
threats to the program or to gain a competitive advantage. Cell c involves the fortuitous 
use of study results. This may occur when an unanticipated opportunity arises to use the 
study results to enhance the status, of the organization or program and/or to gain a 
competitive advantage. Or, the study results are misconstrued (either deliberately or 
unintentionally) so as to reflect negatively on the program or organization. Cell d involves 
the intended or planned use of study results and is the topic to which we shall devote the 
remainder of this chapter. 
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17.2 The Planned Use of Evaluation Results 

Even though one cannot absolutely guarantee that the results of an evaluation study will 
be used (assuming that they are worthwhile) there is much that can be done to facilitate 
the entire process of dissemination and utilization. Two obvious considerations are to plan 
far enough in advance for different kinds of dissemination and utilization and, to involve 
potential users early on (Patton, 1986 ). 

In a system as vast and as complex as the Cooperative Extension System the use of 
results from evaluation studies occurs in many, many different ways rather than being 
* focused on a single decision maker or decision making body. In recognition of this fact 
and in order to reach its' diverse and far flung audiences with the results of Its many 
evaluation studies, the Extension Service of the U.S. Department of Agriculture developed 
the matrix format given in Figure 17.1 . To use this format or one like it, the potential user 
groups for the study results are first identified, then the ways in which they would likely 
use the results and, as a consequence, the form in which the results might be presented 
to them. 

o User Groups - categories of user groups are identified for a study. Not ail of the 
groups listed in the matrix would be appropriate for any one study. A complete list of 
those possible is used so that ail of the studies can be handled in one format. However, 
for a singie broad ranging study, ail of those listed would likely be appropriate. [Some 
might even be broken down into sub-categories.] Most of the user group categories are 
self-explanatory. Extension Staff might include the Extension Committee on Organization 
and Policy with its substantively oriented sub-committees of: strategic planning; 
legislation; budget; program leadership; and, personnel and organizational development 
( mentioned in earlier chapters as important decision-making groups for the system). To 
the extent that an evaluation study has findings of scholarly interest they might be shared 
with Professional Groups and archived in the Libraries as a contribution to the research 
and knowledge base. 

o Types of Utilization - the types* of utilization are also fairly self-explanatory. 
Awareness and Information focuses on "getting the word out" while Program 
Justification is concerned with accountability - is a program doing what it should be 
doing in terms of focus on the problem - the who, where, gets what, when and how 
often, hopefully with the citation of some benefits, even if they are only anecdotal in form. 
Program Improvement is concerned with those aspects of the study's findings that might 
be used to enhance program performance in terms of efficiency, effectiveness or client 
payability ( the "it works but we can't stand it" phenomenon). Program Redirection is 
much as it is outlined in Chapter. 13. Any evaluation results may have Policy 
Implications in terms of the functioning of the program or the functioning of the 



* Shadish et al., (1991) report some theorists as using the concepts of instrumental 
(making changes), conceptual (developing understanding) and persuasion (promoting or 
advocating) to think about types of utilization. However, the ones used in Figure 17.1 
are better suited to the needs of Extension. 
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organization of which it is part or, for that matter, for other related organization's. The 
policy might pertain to criteria of eligibility for participation in the program, kinds of 
experiences that should or should not be provided and by whom, roles that other 
organizations should or shouldn't play, etc. Staff Development is an often overlooked 
use of evaluation results. Evaluation studies may uncover staff training needs or lead to 
the development of new materials for such training. The Research Base is important for 
Extension and one would want to insure that findings that might have relevance for future 
programming be made accessible to a variety of groups. Other is just a residual category 
for concerns that might not be covered in the preceding categories or might involve the 
idiosyncrasies of a particular study. 



o Modes of Dissemination - Some possible modes of dissemination are listed - 
undoubtedly others may also be relevant. The first five (Media Releases, Popular 
Reports, Video Tape/Disc, Cassette/Diskette, Fact Sheets) are all vehicles for getting 
the results out in a fairly succinct form. The next three (Briefings, 
Meetings/Conferences, Speakers Bureau) involve some personalized presentation to 
what may be a fairly "captive" audience by a person who should be able to answer what 
may be fairly detailed questions from them. An Implications Memo teases out the 
implications of the findings for changes in program policy and/or practice. It is a document 
that is usually negotiated by the PE with the program staff and is not usually given wide 
circulation. It is particularly useful when there is a decision making body to which the 
results should be targeted. The Program Profile, as we saw in Chapter 13, contains an 
array of information about a program that is prepared for use by the Life Cycle Guidance 
Team (LCGT) or the Transition Management Team (TMT) for purposes of program 
review, ranking and redirection. Analytic Formats refers to the presentation of results in 
analytic or tabular form and is often used in policy formulation when a decision making 
body wants to explore the implications of different policy options and view their numerical 
consequences, especially in resource allocation kinds of efforts. Finally, for any study a 
Technical Report must be developed in sufficient detail so that other investigators might 
examine them to determine if they could use them for other, similar studies or if they 
could have come up with the same results themselves if they were to repeat the study. 

A matrix like this can be used far_each .study with the cell entries being P for Planned, 
I - In Process or C - Completed; with each up-dated periodically. Hence, a date entry 
in the upper right hand box. 

Obviously the content and tone of the different modes will vary with the user group and 
their anticipated type of utilization. Usually, the PE in cooperation with the program staff, 
will specify the study findings and recommendations and then these will be formatted by 
an information staff who are accustomed to preparing materials for such audiences. 
Critical information is usually included in most but often couched in the language of 
program improvements. 

This same format can be used for State and for Local evaluation results. However, the 
user groups, types of utilization and modes of dissemination might differ considerably. 
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17.3 Examples of Planned and Actual Use 

By way of illustrating some of these concepts we can examine examples of some of the 
"planned and actual uses : a few of the in-depth studies described in the previous 
chapter: 

o Nutrition Education for Low-Income Families - this study was mandated by 
Congress in 1979 in a surprisingly detailed manner entailing one full page of questions 
to be addressed. It arose out of a conflict between the Senate and House Appropriations 
Committees. The former wanted to cut the funding while the latter wanted to preserve it. 
The House prevailed but the study was a kind of compromise position in that they would 
revisit the topic when the study results became available. In the Spring of 1980, with the 
advent of a new administration there was much interest in Congress in defunding what 
were thought to be "social" programs. However, Congressional staff who were supporters 
of the program pointed out that the program was currently toing evaluated and that they 
should wait for the evaluation results before making any decisions. When the study 
results became available in late 1 981 Congress had lost it's fervor for such programs and 
moved on to other topics. Study results were favorable for the program and were 
presented through a series of briefings to Congressional, USDA and Extension staff, via 
a brief written report to Congress and a one page fact sheet on the study procedures and 
results was widely disseminated throughout government and throughout the Cooperative 
Extension System. A deliberative body representing State and Federal Extension staff 
was formed by the ES Administrator to review every thing that was known about the 
program including the study results and make recommendations for change. As a result 
efforts were initiated to revise the reporting system and to update and standardize the 
curriculum. 

o 4-H Youth Development: Alumni - this study was requested by a 4-H Impact 
Study Committee and was intended to take a retrospective look at how 4-H alumni as well 
as Alumni of other youth organizations felt that they benefited from their participation and 
to gauge the extent of their involvement in current community affairs. Summary reports 
and one page fact sheets were given wide circulation both within and outside of 
Extension. A video tape about the results was developed and made available to each 
State so that they in turn might make copies available to their counties. An Implications 
Memo was developed through negotiations with the Impact Study Committee proposing 
specific policy and programmatic changes. Upon completion, copies of the Memo were 
shared with all States accompanied by a cover memo endorsing the recommendations 
signed jointly by the ES/USDA Deputy Administrators for 4-H Youth Development and 
for Planning, Development and Evaluation. 

o Extension and Volunteers - this study was requested by the then ECOP 
subcommittee on Accountability and Evaluation. Among other findings this study showed 
that Extension works with about 2.9 million volunteers annually thereby multiplying five- 
fold the combined Federal-State-local contributions and enabling Extension to reach some 
48 million persons each year. [ This number, rounded to 3 million, has now become part 
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of the lore of Extension and appears in many of its' publications and presentations.] Study 
results were presented to Congress as part of the appropriations process, to the public 
and clientele via media releases and one page fact sheets (the latter also for CES and 
USDA staff) and through a series of briefings to USDA staff. In addition, presentations 
were made at professional meetings . A video tape of the results was made available to 
every State and county and a complete set of the reports was housed in each Land-Grant 
University library. Many States used some of these reports in developing or revising their 
training materials for both staff and volunteers; and, the results were used as part of an 
effort to develop a system-wide strategic plan for work with volunteers and leadership 
development. 

o Contemporary Youth and Contemporary 4-H - this study was requested by 
a 4-H Impact Study Committee and was initiated to test the feasibility of conducting a 
longitudinal study of youth including those in 4-H via a phone survey (at least initially). 
Response rates for non-4-H youth were found to be unacceptably low and potential study 
costs too high. Hence, the plan was abandoned and as an alternative, secondary 
analyses were performed on longitudinal data available from the U.S. Department of 
Education which had a limited amount of information on youth participation in 4-H and 
other activities. These analyses showed that the program was very different from its' past 
with dramatically more minority youth being served in in-school and after class settings' 
(Steele, et al., 1993a; 1993b). 

o Leadership Development - requested by the then Accountability & Evaluation 
subcommittee of ECOP, the results from this study were presented via a series of 
briefings to CES staff and a non-technical summary report was given wide circulation both 
within and outside of Extension. The results were used by local staff to better define the 
program and its' activities and were also used as part of an effort to develop a system- 
wide strategic plan for leadership and volunteer development. 

o Youth at Risk -this study was initiated at the request of the ES/US^A Deputy 
Administrator for Management who wanted to know how well CES staff were at working 
with youth from high risk environments. An outside contract firm was hired to do the study 
with their results being provided via a series of briefings and a technical report. ES 
developed a one page brochure on the study's procedures and findings which was widely 
disseminated to CES staff and stakeholders. As a result of the study some projects were 
refocused to deal more intensively with at risk youth. 

o Polish/American Extension Project - this study was requested by the three 
agencies involved (CES, USAID, FAO) to see to what extent such an effort was 
worthwhile and whether it should be continued or even fostered in other former Eastern 
bloc countries. Study results were presented in a series of briefings and a written report 
whose Executive Summary was given wide distribution. The results were in the affirmative 
and funding was renewed with certain project efforts being strengthened. 
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o Nationwide Extension System - requested by the then Accountability and 
Evaluation subcommittee of ECOP, the results from this study were presented via a 
series of briefings to USDA staff, a technical report and a one page fact sheet which was 
given wide circulation. The results were used by a committee especially formed by ECOP 
to further define and delineate the Federal-State partnership in Extension. 

o Extension/Research Complex - this study resulted in the development of a new 
conceptual model of how the CES can and should function within this larger complex with 
a series of recommendations as to how this can be accomplished. The results were 
presented via a series of briefings or presentations to CES, USDA and staff of other 
organizations/groups, through a series of brief summaries some of which appeared in 
special journals or publications and, a book length report of the entire effort. These results 
have been and continue to be used to help articulate the unique roles that Extension 
plays (or might play) in this larger complex. 



ERIC 



17 " 7 342 



REFERENCES 

Atfholter, D. (1993) Latter to the Editor. Evaluation Practice 14(3). 

Albrecht, K.; & D. Deshler. (1990). Managing change and transition. Raleigh, NC: North Carolina 
State University Extension Service. 

Allen, W. & E. Rajotte (1987) The national evaluation of Extension's Integrated pest management 
(IPM) programs. Blacksburg, VA.: Cooperative Extension Service, Virginia Tech. 

American Evaluation Association Draft guidelines for evaluators .(1993) Association Publication. 

American Psychological Monitor. November, 1993. 

Anderson, B. & S. Nilson (1964). Studies In the reliability and validity of the critical Incident 
technique. Journal of Applied Psychology. 48 (pp.398-403). 

Armstrong, J. (1985). Long-range planning. New York, NY., Wiiey. 

Bandura, A. (1986). Social foundations of thought & action. Engtewood Cliffs, N.J.:Prentice-Hall, Inc. 

Bandura, A. (1988). Organisational applications of social cognitive theory. Australian Journal of 
Management . 13(2). 

Bandura, A., (1991). Social cognitive theory of self-regulation. Organizational Behavior and Human 
Decision Processes . Vol. 50 (pp.248-287). 

Bartholic, J.; F. Fear & J.Lichty. (1987) Agricultural weather Information systems. East Lansing, ML: 
Michigan State University. 

Beaton , A. (1969) Criterion scaling of questionnaire items for regression analysis. Socio- 
Economic Planning Sciences, (2) 335-362. 

Beaton, A. & G. Mayeske. (1986b). Testing the effectiveness of pseudo-longttudlnal study designs 
with systematic attrition. Paper presented at the 21st International Congress ol Applied Psychology, 
Jerusalem, Israel. 

Beaton, A. & G. Mayeske. (1986a). A pseudo-longttudlnal model for cross-cultural analyses. Paper 
presented at the 8th International Congress of Cross-Cultural Psychology, Istanbul, Turkey. 

Beaton, A. (1982). Approximating mean longitudinal curves using cross-sectional data. 
Unpublished manuscript. Princeton, NJ. 

Beaton, A. (1986). Primer for use of pseudo-longttudlnal growth model. Unpublished Manuscript. 
Princeton, NJ. 

Beaton, A. & G. Mayeske. (1984). How long should a longitudinal study be?. Paper presented at the 
Interamerican Congress of Psychology, Acapulco, Mexico. 

Beaton, A. (1981). Approximating longitudinal analysis from cross-sectional data. Unpublished 
manuscript. Princeton, NJ. 



18-1 

343 



Beaton, A. (1983). A pseudo-longltudlnal mod*! with drop-out adjustment. Unpublished manuscript. 
Princeton, NJ. 

Beaton, A. (1975) Pseudo-longltudlnal model. Unpublished manuscript. Princeton, NJ. 

Benedict, R. (1934). Patterns of culture. Boston: Houghton-Mifflin. 

Bennett, C. (1990). Cooperative Extension roles & relationships for a new ere (summary & 
technical report). Washington, D.C.: Extension Service, U.S. Department of Agriculture. 

Bennett, C. (1979). Analyzing impacts of extension programs. Washington, DC: U.S. Department of 
Agriculture, Science & Education Administration (ES C-575). 

Bennett, C. (1975). Up the hierarchy. Journal of Extension 13(2) pp. 7-12. 

Bennett, C. (1992). Cooperative Extension roles & relationships for a new ere (brief summary). 

Washington, D.C.: Extension Service, U.S. Department of Agriculture. 

Benventste, G. (1989). Mastering the politics of planning. San Francisco, CA: Jossey-Bass. 

Bickman, L (1987). Using program theory In evaluation. New Directions for Program Evaluation, no. 
33. San Francisco, CA.: Jossey-Bass. 

Bickman, L (1990). Advances In program theory. New Directions for Program Evaluation, no. 47. 
San Francisco, CA.: Jossey-Bass. 

Black, D., G. Howe; D. Howell & P. Becker (1992) Selecting advisory council members. Journal; of 
Extension. Vol.30(1). 

Boone, E. (1985). Developing programs In adult education. Englewood Cliffs, NJ.: Prentice-Hall, Inc. 

Bottum, J. (1993) An assessment of state funding for the cooperative extension system. Wash, 
DC: Extension Service, U.S. Department of Agriculture. 

Boyle, P. (1981). Planning better programs.New York,N.Y.: Mcgraw-Hill, inc. 

Brannon, E. & Evans, D. (1987). Extension's small home based business programming In 
Alabama: an evaluablllty assessment. Auburn, AL: Alabama Cooperative Extension Service, Auburn 
University. 

Brazzel, M.; & D. Sanderson. (1990). Managing cooperation: Implementing Issues programming In 
the cooperative extension system. Raleigh, NC: North Carolina State University Extension Service. 

Brewer, J. & A. Hunter. (1989). Multbnethod research. Newbury Park, CA.: Sage. 

Brown, T. & L. Hale (1988a) Financial planning & management programs: comprehensive report. 

Columbia, MO.: University of Missouri Cooperative Extension Service. 

Brown, T. & L Hale (1988b) National Impact study - family financial planning & management 
programs. Columbia, MO.: University of Missouri Cooperative Extension Service. 

Bryson, J. (1989). Strategic planning for public and non-profit organizations. San Francisco, CA.: 
Jossey-Bass. 



ERLC 



18-2 

344 



Carroll, J. & E. Johnson. (1990). Decision research. Newbury Park, CA,: Sage. 

Champion, J. & JJones (1975) .Critical Incidents In management. Homewood, IL: Richard D. Irwin, 
Inc. 

Chen, H. (1990). Theory driven evaluations. Newbury Park, CA.: Sage Publications. 

Claik, C. (1990). An evalu ability assessment of the planning & reporting system of the Illinois 
Cooperative Extension Service. Urbana, IL.: Illinois Cooperative Extension Service, University of 
Illinois. 

Cleland.D. & W.King(eds)(1988).Project Management Handbook.New York: Van Nostrand Reinhold. 
Coates, J. (1986). Issues management. Mt. Airy, MD.: Lomond Publications, Inc. 

Cole,J. & M. Cole.(1983). Advisory councils: a theoretical 4 practical guide for program 

managers. Englewood Cliffs, N.J.: Prentice-Hall. 

Coleman, J. et at, (1966) Equality of educational opportunity. Wash., D.C.: U.S. Govl Printing Office 
#FS 5-38001. 

Connors, K. (1992). How DO we define need? Needs Assessment Notes 4(2) of the Topical Interest 
Group on Needs Assessment of the American Evaluation Assn. 

Corwin, R. (1982). The entrepreneurial bureaucracy: biographies of two federal programs. 

Greenwich, CT.: JAI Press. 

Davis, W. & S. Verma (1993). Performance appraisal: how Extension agents view the system. 

Journal of Extension XXXI (Winter). 

Denzin, K. (1989). Interpretive Interactlonlsm. Newbury Park, CA.: Sage. 

Donohue, J. (1994) The fat cat freeloaders. The Washington Post Outlook section (3/6/94). 

Douglas, J. (1976). Investigative social research. Beverly Hills, CA.: Sage. 

Etling, A. & S. Tuttle. (1988). An evaluablllty assessment of Pennsylvania's 4-H program. State 
College, PA.: Pennsylvania State Cooperative Extension Service, Pennsylvania State University. 

Evaluation Practice News October; 1993: 

Fetterman.D.M. (1989). Ethnography :step by step. Newbury Park,CA.:Sage. 

Flanagan, J. (1954). The critical Incident technique. Psychological Bulletin 51(4). 

Gagne', R. (ed.) (1987). Instructional technology foundations. Hillsdale, NJ.: L. Erlbaum Ass. Inc. 

Gagne'.R.; L. Briggs; W. Wager. (1992). Principles of Instructional design. Orlando.FL: Harcourt, 
Brace, Jovanovich College Publishers. 

Guba, E. & Y. Lincoln. (1989). Fourth generation evaluation. Newbury Park, CA., Sage. 



18.3 345 



9 

ERIC 



Qutierres, J. (1988). Use of evaluabillty assessment to plan Hawaii's middle management 4-H 
program. Manoa, HI.: Hawaii Cooperative Extension Service. 

Hahn. C; G. Brumbach; & D. Edwards (1979). Development of a performance evaluation system. 

.Wash., DC: U.S. Dept. Agriculture by the American institutes for Research. 

Hammond, D. (1989). Evaluabillty assessment of retail trade programs. Ames, IA.: Iowa State 
University Extension, Iowa State University. 

Hastings, S. (1986). Evaluabillty assessment of the Mississippi 4-H program. Starkville, MS.: 
Cooperative Extension Service, Mississippi State University. 

Henry, G. (1990). Practical sampling. Newbury Park, CA.: Sage. 

House, E. (1993) . Professional evaluation: social Impact & political consequences. Newbury Park, 
CA.: Sage 

James, F. (1991). Local foods make your body healthy: promoting high nutrient density produce 
in the CNMI. Saipan, CNMI.: Northern Marianas College, School of Agriculture & Life Science. 

Johnson, D. et at, (1987). Needs assessment theory and methods. Ames, IA.: Iowa State University 
Press. 

Johnson, E. & R. Richard. Influence on subsequent behavior of evaluabillty assessment workshop 
participants. (1993) Paper presented at the annual meetings of the American Evaluation Assn., 
Dallas. TX. 

Jones, J. (1992). Teaching Clientele what or how to think. Journal of Extension (xxx). 

Jorgensen.D.L. (1989). Participant observation. Newbury Park,CA:Sage. 

Kelly, J. & J. McGrath. (1988). On time and method. Newbury Park, CA: Sage. 

Kerzner, H. (1989). Project management. New York, NY.: Van Nostrand ReinhokJ. 

Kerzner, H. (1984). Project management: a systems approach to planning, scheduling and 
controlling. New York, NY.: Van Nostrand ReinhoW Co., Inc. 

Kettner, P.; R. Moroney; & L. Martin. (1990). Designing & managing programs. Newbury Park, CA.: 
Sage. 

Kezsbom, D.; D. Schilling; & K. Edward. (1989). Dynamic project management. New York, NY.: 
Wiley. 

Killam, D. (1990). Evaluabillty assessment of Maine's leadership development programs. Orono, 
ME.: Cooperative Extension Service, University of Maine. 

Killam, D. (1989). Evaluabillty assessment of Maine's training programs for the elderly. Orono, 
ME.: Cooperative Extension Sen/ice, University of Maine. 

Killam, D. (1989). Evaluabillty assessment of Maine's child care training programs. Orono, ME: 
Cooperative Extension Service, University of Maine. 



— 346 



Krippendorf, K. (1980). Content analysis. Beverly Hills, CA.: Sage. 
Krueger, R. (1988). Focus groups. Newbury Park, CA.: Sage. 

Ladewig, H.; & J. Thomas. (1987). The national 4-H alumni study: assessing the Impact of 4-H on 
former members. College Station, TX.: The Texas A&M University System. 

Ladewig, H. (1994). Personal communication at the March GPRA/Extenslon Pilot Project 
meetings, Denver, CO. 

Larrtur, M. & J. Burtner. (1993). Moving to higher ground project. Blacksburg, VA.: Virginia 
Cooperative Extension Service, Virginia PI &SU. 

Larrtxjr, M. & K. Stephenson. (1990). An evaluablllty assessment of Virginia cooperative extension 
service water quality programs. Blacksburg, VA: Virginia Cooperative Extension Service, Virginia 
Polytechnic Institute & State University. 

Lee, I. (1991). Use of evaluablllty assessment to plan for families with young children at risk. 

Pine Bluff, AR. (1890 Consortium), Arkansas Cooperative Extension Service. 

Leinhardt, J. (1989). Evaluablllty assessment of Arkansas' economic development efforts. Little 
Rock, AR.: Cooperative Extension Service, University of Arkansas. 

Levin, H. (1983) Cost effectiveness: a primer. Newbury Park, CA.: Sage. 

Lites, R.; & D. Mustian. (1990). Programming processes applied to Issues programming. Raleigh, 
NC: North Carolina State University Extension Service. 



Lippke,L; H. Ladewig & E. Taylor-Powell (1987) National assessment of extension efforts to 
Increase farm profitability through Integrated programs. College Station, TX.: Texas A&M 
University Agricultural Extension Service. 

Louis, K.; & S. Sieber (1979). Bureaucy & the dispersed organization: the educational extension 
agent experiment. Norwood, NJ.: Ablex Publishing Co. 

Lovan, R. (1988). Using evaluablllty assessment for the ES/USDA rural revltalizatlon plan of woric. 

Washington, DC: Extension Service, U.S. Department of Agriculture. 

Mager, R. (1972). Goal analysis. Belmont, CA.: Ritman Learning Inc. 

Mager, R. (1975). Preparing Instructional objectives. Belmont, CA.: Fearon Publishers Inc. 

Majchrzak, A. (1984). Methods for policy research. Newbury Park, CA,: Sage. 

Makus, L; J Carlson & R. Krebill-Prather (1989) An evaluation of the futures & options marketing 
pilot program. Moscow, ID. University of Idaho, Departments of Agricultural Economics & Rural 
Sociology. 

Mayberry, B. D. (1989). The role of Tuskegee In the origin, growth and development of the Negro 
cooperative extension system 1881-1990. Montgomery, AL: Brown Printing Co. 



18-5 

347 



Mayeske, G. & A. Beaton (1975) Special studies of our nation's students. Washington, D.C.: U.S. 
Govl Printing Office #01 7-080-01 380-1 . 

Mayeske, G. (1969). The development of a model for student achievement. Socio-Economic 
Planning Sciences , 2 (pp. 363-371). 

Mayeske, G.; T. Okada; W. Cohen; A. Beaton; & C. Wisler (1973a). A study of the achievement of 
our nation's students. Washington, D.C.: U.S. Gov't Printing Office #1780-1055. 

Mayeske, G.; T. Okada; & A. Beaton (1973b) A study of the attitude towards life of our nation's 
students. Washington, D.C.: U.S. Govt Printing Office #1780-01 146. 

Mayeske, G.; C. Wisler; A. Beaton; F. Weinfekf; W. Cohen; T. Okada; J. Proshek; & K. Tabter. (1972) A 
study of our nation's schools. Wash., D.C.: U.S. Govl Printing Office # HE 5.210:10085. 

Mayeske, G. (1982) An assessment of the objectives, Implementation & effectiveness of the 
Expanded Food & Nutrition Education Program. Wash, DO: Extension Service, U. S. Department of 
Agriculture. 

Mayeske, G. (1966). What can critical Incidents tell management? Training & Development Journal 
20(4). 

Mayeske, G. (1990). The value of a nationwide extension system. Washington, D.C.: Extension 
Service, U.S. Dept. Agriculture. 

Mayeske, G & A. Glickman. (1964). Recall of critical Incidents. Personnel Research Series. U.S. 
Department of Agriculture, Office of Personnel PRS 64-2 Wash. D,C. 

Mayeske, G. (1992). Evaluablllty assessments of five rural economic development programs: a 
synthesis. Washington, D.C.: Extension Service.U.S. Dept. of Agriculture. 

Mayeske, G. (1991). Program design : an evaluation approach, (vols. 1,2, & 3). Washington, D.C.: 
Extension Service, U.S. Dept. of Agriculture. 

Mayeske, G. (1966). Critical requirements of performance for auditors and special agents. The 

Internal Auditor. Fall. 

McKillip, J. (1987). Need analysis. Newbury Park, CA: Sage r 

McMillan, I. (1983). Competitive strategies fornot-for-proftt agencies. Advances in Strategic 
Management 1 , 61-82, JAI Press Inc. 

Meister, G. (1989). Evaluablllty assessment report: the managing main street business program. 

Lincoln, NE.: University of Nebraska-Lincoln Cooperative Extension. 

Michael, J. (1989) Developing leadership among extension clientele: an evaluation of Inputs. 

Wash, D.C.: U.S. Department of Agriculture Extension Service in cooperation with Washington State 
University. 

Mood, A. (1983) Introduction to policy analysis. New York, N.y.: Elsevier Science Publishing Co., Inc. 
Moore, C. (1987). Group techniques for Idea building. Beverly Hills, CA.: Sage. 



ERIC 



Morgan, D. (1988). Focus groups as qualitative research. Newbury Park, CA.: Sage. 

Morrison, J.; W. Renfro & W. Boucher. (1984). Futures research & the strategic planning process: 
Implications for higher education. Washington, DC: ASHE ERIC Higher Education Research Report 
. no. 9., Association for the Study of Higher Education. 

Mortvedt, M. (1990). Evaluablllty assessment of the planning & reporting system. Manhattan, KS.: 
Kansas Cooperative Extension Service. 

Mortvedt, M. (1991). Evaluablllty assessment: community development economic development. 

Manhattan, KS.: Kansas Cooperative Extension Service. 

Mueller, M. R. (1991). Quality means results. Journal of Extension, Vol. XXIX. Spring, 1991 . 

Mustian, R. (1993). Area specialists/agents: program design. Raleigh, N.C.: Cooperative Extension 
Service, NC Sate Univ. 

Muther, R. (1988). High performance planning: a basic approach to planning. Kansas City, MO.: 
Management & Industrial Research Publications. 

Nelson, P. (1987). Evaluablllty assessment on a shoestring: Delaware's experience with the 
family well-being plan of work. Newark, DE: Cooperative Extension Service, University of Delaware. 

Pancer, S. & A. Westhues. (1989) A developmmontal stage approach to program planning & 
evaluation. Evaluation Review .13(1) pp. 56-77. 

Parersti, M. (1988). Democracy for the few. New York: St. Martin's Press, Inc. 

Patton, M. (1990). Qualitative evaluation & research methods. Newbury Park, CA.: Sage. 

Patton, M. (1986). Utilization • focused evaluation. (2nd ed.) Beverly Hills, CA.: Sage. 

Phillip, J. (1991). Proceedings of evaluablllty assessment workshops on production, marketing 
and utilization of local produce. Kolonia, Pohnpei: Cooperative Extension Service, Land Grant 
Programs, College of Micronesia. 

Prochaska, J., C. DiClemente & J. Norcross. (1992). In search of how people change: applications 
to addictive behaviors. American Psychologist 47(9), September, 1992. 

Rasmussen, W. (1989). Taking the university to the people: seventy five years of cooperative 
extension. Ames, IA.: Iowa State University Press. 

Reichardt, C. & S. Rallis (eds.) (1994)The qualitative-quantitative debate. New Directions for Program 
Evaluation (61) San Francisco: Jossey-Bass. 

Reiser, R. & R. Gagne'. (1983). Selecting media for Instruction. Englewood Cliffs, N.J.: Educational 
Technology Publications. 

Richard, R. & E. Miller. (1992). Evaluablllty assessment of the Louisiana 4-H youth development 
program. Baton Rouge, LA.: Cooperative Extension Service, Louisiana State Univ. 

Rockwell, K. (1991) Organization & Implementation assessment of the FY 90-94 water quality 
demonstration projects. Lincoln, NE.: University of Nebraska Cooperative Extension. 



18-7 349 



Rogers, E. (1983). Diffusion of Innovations. NY. The Free Press 

Rogers, E. & I. Kincaid. (1982). Communication networks. NY.: The Free Press. 

Ronan, W. & G. Latham (1974). The reliability and validity of the critical Incident technique. 

Studies in Personnel Psychology 6(1) pp.53-64. 

Rosenthal, R. (1984). Meta-analytic procedures for social research. Newbury Park, CA.: Sage. 

Rossi, P. & H. Freeman. (1985). Evaluation: a systematic approach (3rd ed.).Beverly Hills, CA.: 
Sage. 

Ruchelman, L. (1985). A workbook In program design for public managers. Albany, NY, State 
University of New York Press. 

Russell, M. (1990). Evaluablllty assessment of the Connecticut water quality plan. Storrs, CT.: 
University of Connecticut Cooperative Extension. 

Russell, M. (1989). Evaluablllty assessment of the 4-H youth development program. Storrs, CT.: 
University of Connecticut Cooperative Extension. 

Sadowske, S. (1991). Toward futuristic program planning In adult education^ thesis submitted in 
partial fulfillment of the requirements for the degree of Doctor of Philosophy in Continuing & Vocational * 
Educatk>n).Madison,WI.: University of Wisconsin. 

Schwandt, T. & E. Halpern. (1988). Linking auditing & metaevaluatlon. Newbury Park, CA.: Sage. 

Scriven, M. (1993). Hard - won lessons In program evaluation. New Directions for Program 
Evaluation. No. 58, Summer, 1993. 

Scriven, M. (1991). Evaluation thesaurus (4th ed.) Newbury Park, CA.: Sage 

Scriven, M. & J. Roth (1978). Needs assessment: concept & practice. New Directions for Program 
Evaluation, 1, 1-11. 

Sechrest, L. (ed.) (1993) Program evaluation: a pluralistic enterprise. New Directions for Program 
Evaluation (60). San Francisco: Jossey-Bass. 

Sechrest, L, J. Babcock & B. Smith (1993) An Invitation to methodological pluralism. Evaluation 
Practice 14(3). 

SedgewSck, R. (1983) Algorithms. Reading, MA.: Addison Wesley. 

Shadish, W.,T. Cook, & L. Levtton (1991). Foundations of program evaluation. Newbury Park, CA.: 
Sage. 

Sieber, S. (1981). Fatal remedies. New York, NY.: Plenum Press. 

Sikes, W.; A. Drexler & J. Grant. (1989) The emerging practice of organization development. 

Alexandria, VA.: NTL Institute for Applied Behavioral Science. 



ERLC 



18-8 



350 



Simerly, R. & Associates. (1987). Strategic planning and leadership In continuing education. San 

Francisco, CA.: Jossey-Bass. 

Sims, H. & P. Lorenzi. (1992). The new leadership paradigm: social learning & cognition In 
organizations. Newbury Park, CA.: Sage. 

Smith, M.F. (1989a). Evaluablllty assessment: a practical approach. Boston, MA.: Kluwer. 

Smith, N.L. (ed.) (1992) Varieties of Investigative evaluation. New Directions for Program Evaluation, 
#56, San Francisco, CA.: Jossey-Bass, Inc. 

Smith, R. (1989) National assessment of producer marketing ahernatlves. College Station, TX.: 
Texas A&M University Agriculture Extension Service. 

Smith, M.F. (1992). Criteria for excellence: relevance, quality, utility. College Park, MD.: 
Cooperative Extension Service, Univ. of Maryland. 

Smith, M. F. (1991). Criteria forjudging excellence. Journal of Extension. Vol. XXIX. Spring, 1991. 

Smith, M.F. (1989b). Evaluablllty assessment of Maryland water quality/quantity program. College 
Park, MD.: Cooperative Extension Service. 

Steele, S.; C. Finley; & C. Edgerton. (1989). Partners for action: the roles of key volunteers. 
Madison, Wl.: University of Wisconsin, Cooperative Extension. 

Steele, S. (1989) Contemporary youth & 4-H: test of a study design. Madison, WL: University of 
Wisconsin Cooperative Extension Service. 

Steele, S.; T. Miller; K. Rai; M. Appel & R. Jones. (1993b)Gettlng acquainted with U.S.elghth 
grader8:lmpllcatlons for nonschool programs.Madison, Wl.: Dept. of Continuing & Vocational 
Education/Cooperative Extension, University of Wisconsin - Madison. 

Steele, S.; T. Miller & K. Rai. (1993a). Nationwide participation In 4-H during the 1980*8. Madison, 
Wl.: Dept. of Continuing & Vocational Education/Cooperative Extension, University of Wisconsin - 
Madison. 

Stewart, D.; & P. Shamdasani. (1990). Focus groups: theory & practice. Newbury Park, CA.: Sage 

Stuff lebeam, D. (1981). Standards for evaluations of educational programs, projects & materials. 

NY. McGraw-Hill. 

Stufflebeam, D. et al., (1988). The personnel standards. Newbury Park, CA.: Sage 

Suchman, E. (1967). Evaluative research. New York, NY.: Russell Sage Foundation. 

Summers, J. (1989). Evaluablllty assessment of Missouri's economic/community development 
efforts. Columbia, MO.: Extension Service, University of Missouri. 

Sylvia, R., K. Meier, & E. Gunn. (1985). Program planning & evaluation for the public manager. 
Monterey, CA.: Brooks/Cole. 

TASCON (1993) An assessment of the Cooperative Extension System's national Initiative on 
youth at risk. Bethesda, MD.: The TASCON Corporation 



9 

ERJC 



18 ■ 9 

351 



Taylor-Poweil, E.; & L Lippke. (1990). The role of disciplines In Issues programming. Raleigh, NC: 
North Carolina State University Extension Service. 

Tesch, R. (1992) P re-sesslon on Qualitative Analysis of the 1992 American Evaluation 
Association meetings, Seattle, WA. 

Tyler, R. (1971). Basic principles of curriculum and Instruction. Chicago: University of Chicago 
Press. 

United Way of America. (1982). Needs assessment. Alexandria,VA. 

Van den Ban, A. & Hawkins, H. (1988). Agricultural extension. New York, NY.: Longman Scientific & 
Technical/Wiley. 

Van Gundy, A. (1988b). Techniques of structured problem solving. New York: Van Nostrand 
Rheinhold Co. Ltd. 

Van Gundy, A. (1988a). Stalking the wild solution: a problem finding approach to creative 
problem solving. Buffalo, N.Y.: Bearty Limited. 

Verma, S. (1991). Evaluablllty assessment of Louisiana's leadership development programs. 

Baton Rouge, LA.: Louisiana Cooperative Extension Service, Louisiana State University. 

Verma, S. (1990). An evaluablllty assessment of Louisiana cooperative extension service water 
quality program. Baton Rouge, LA: Louisiana Cooperative Extension Service, LSU. 

Wade, W. (1993). Adolescent pregnancy & parenting: program design. Greensboro, N.C.: 
Cooperative Extension Sendee, NCA&T University. 

Warner, P & J. Christenson (1984) The cooperative extension service: a national assessment. 

Boulder, CO.: Westview Press. 

Webster's 7th New Collegiate Dictionary (1963) Springfield, MA.: G&C Merriam Co. 

Weick, K. (1976). Educational organizations as loosely coupled systems. Administrative Science 
Quarterly 21: 1-19. 

Weisbord, M. (1987) Productive workplaces. San Francisco, CA.: Jossey-Bass, Inc. 

Wessel, T. & M. Wessel (1982). 4-H, air American Idea; 1900-1980: a history of 4-H. Chevy Chase 
MD.: National 4-H Council. 

West, D.; H. Bahn & R. Mayer (1993 ) Assessment of the Polish/American extension project. 
Wash, DC: Extension Service, U.S Department of Agriculture. 

Wholey, J. (1983). Evaluation & effective public management. Boston, MA.: Little, Brown & Co. 
Wholey, J. (1979). Evaluation: promise & performance. Washington, DC: The Urban Institute. 
Whyte,W.F.(ed.). (1991). Participatory action research.Newbury Park,CA.:Sage. 
Wisman, J.(ed.). Worker empowerment. New York: The Bootstrap Press. 



18-10 

35£ 



Workman, R. (1988). Marketing education designed for calculated Impact on Guam: using 
evaluabllity assessment for program planning. Mangilao, GUAM.: College of Agriculture & Life 
Sciences, University of Guam. 

Workman, R. (1990). Using evaluabllity assessment to plan the community resource development 
program. American Samoa: Land Grant Program, American Samoa Community College. 

Yin, R. (1989). Case study research: design & methods. Newbury Park, CA.: Sage. 

Zimmer, B. & Smith, K. (1992). Successful mentoring for new agents. Journal of Extension (xxx). 



18 - ,1 353 



Appendix A 



Workshop Formats and Materials 
for the Program Design Team Facilitator(s) 



HOW MUCH TIME IS NEEDED: 

Usually a workshop is conducted in a series of 2-2 day sessions. So as to allow the 
participants time to travel to and from their home base, the sessions usually run from 
noon of one day to noon of the third day. However, if travel time is not a problem then 
each session can be held on two full consecutive days even though fatigue of the 
participants and the facilitators) may become a consideration. 

Usually a period of 2-3 months is required between sessions to allow adequate time for 
scheduling, conducting and transcribing the interviews and doing some preliminary 
analyses of the results. This period may vary from a low of 6 weeks to a high of 4 
months depending upon the need for the results or the interference from other events (a 
longer period of time runs the risk of losing the interest of the group). 

Sometimes a third (or even fourth) one-to-two day session is needed to finish up and/or 
to conduct a model verification exercise. Usually if the topic being dealt with is quite 
complex or if the participants have never worked together on such a topic then another 
session(s) may be needed. A model verification exercise is one in which program 
providers who have not been part of the workgroup are brought in to critique the model 
as to how reflective it is of their own experience and suggest appropriate modifications. 
Usually a verification session requires only half to three quarters of a day. 

WHO ATTENDS: 

It is important to have a preponderance of program providers who impact directly on 
clientele in the workgroup for they in a sense are the "reality filters" as to what is or might 
be "doable." Add to them some- administrative staff and some subject-matter specialists 
in order to "round out" different aspects of topical concern. Experienced volunteers may 
also be included, if appropriate. Finally, some one person needs to be named as the 
contact for seeing that arrangements are made, things keep moving, documents get 
analyzed and a report is written. If this person happens to have some training or a 
background in evaluation it can be a real advantage in carrying out some of the technical 
aspects of the work. A group can vary in size from a low of 5 to 6 to a high of about 1 5. 
However, since the process fosters much interaction, more time may be required with a 
larger group. Participants running in and out during sessions can be very 
counterproductive. So too can absentees. 
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WHAT MATERIALS ARE NEEDED: 

Flip charts, magic markers, masking tape and a great deal of wall space is needed to put 
up sheets of paper. A chalk board or its equivalent can also be very useful as long as 
what is on it is written down before it is erased. 

Very little note taking is required since the sheets provide a written record. Some 
however, have still opted to take notes in order to help them prepare the report. Visual 
materials are not used. The configurations of many rooms in which one finds themselves 
are often not conducive to visual presentations. Then again, there may be a lack of 
dependable equipment. Hence the emphasis on a handout which also has the advantage 
of giving the participants some relevant materials to make notes on and take away with 
them. 

The configuration of furniture in the room needs to be such that the design team 
members can readily see and talk with one another as well as with the Facilitator. 
Usually a rectangular or semi-circular seating arrangement with the Facilitator working in 
front of the group works well. 

WHAT ATTRIBUTES AND SKILLS DO FACILITATORS NEED: 

As the Facilitator(s) of the process you should be a third party-to the topic and also to 
the members of the team. Your neutrality and lack of expertise with regard to the topic 
is important for a number of reasons: (1) you do not want to have nor give the 
appearance of having a particular viewpoint; (2) your lack of expertise allows you to ask 
"dumb" questions which can often reveal hidden assumptions or the line of reasoning of 
different team members; and, (3) your naivete' forces members to talk in your layman's 
language rather than their specialized language. You want to be a third party to the 
members of the team also because as Facilitator you need to tend to the tasks on the 
agenda and sometimes force a weary group through the steps involved, tactfully trying 
to avoid conflicts among the members or quiet an unusually loquacious, member. In 
addition, they dont have any views or knowledge of you that is likely to influence them 
in counter-productive ways. It is very difficult for one facilitator to do it all. For the lone 
facilitator fatigue may build up to such a point that the "brain turns to putty," the quality 
of your discourse deteriorates and you -may even find yourself violating some of your 
modeling principles (e.g., putting a dependent event before its precursor, etc.). Hence, 
it is most desirable to have two persons who take turns doing the facilitating. 

The quality of the end products will depend in large measure on how you conduct yourself 
as a Facilitator. You are the guest of a group of otherwise very busy people who may 
feel stretched thin and stressed by all they have to do. You are also their taskmaster for 
the tasks at hand. When team members ask if there is any preparation they have to do 
prior to the workshop sessions we tell them to get a good nights rest and bring a clear, 
fresh brain to the sessions. This is well received and realistic as well, for chances are 
that they wouldn't have time to prepare anything if they had to. There are at least three 
critical skills and a host of 
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desirable attitudes for good facilitating. The skills are the probe, the restatement, and 
the delay. The proba is characterized by such phrases as: "say some more about that;" 
"run that one by me again;" "excuse me for being dense but I just don't understand;" or, 
"could you explain that to me again." The restatement is characterized by phrases that 
precede an actual restatement or paraphrase what is being said, such as: "dont let me 
put words in your mouth but do you mean....;" "are you actually saying that....;" or, looking 

at another team member and saying "does she mean that ". The delay is more a 

physical stance than it is verbal. The facilitator keeps hesitating to write something on 
the flip chart and maintains a quizzical facial expression. Members may ask "why aren't 
you writing that down?" to which the facilitator may respond "it doesnt sound quite right" 
or "it sounds odd or peculiar:" Sometimes humor develops around when and what the 
facilitator writes down. Remarks such as "we must have gotten the right answer for he 
wrote it down" or "hey, he actually wrote it down the way I said it" are not uncommon. 
All three skills are important for building understanding and consensus among the team. 
Anything that fosters humor and occasional laughter among the group will help lighten the 
burden of the tasks for though seldom boring, the work can be tiring. A few tasteful jokes 
and a few jokers on the team can do wonders. 

Finally, the group needs to work in a climate of mutual tiust, candor and cooperation. If 
status differentials within the group impede its functioning then the group needs to be 
restructured or the matter dealt with openly by you. An unusually talkative or domineering 
member may have to be graciously "muzzled" by pointing out that you would really like 
to have more participation by the others. In an extreme case they might even have to 
be seated differently or removed from the group. A team member may be recalcitrant 
because they have antipathy for planning or feel that it will encroach upon their 
autonomy-the latter sometimes cloaked in academic freedom. You may have to get 
them off the team. In the right climate a bit of "letting down of hair" can occur which can 
be beneficial to both program and interpersonal development and can persist beyond the 
life of the design team. 

Sometimes team members will ask "how are we doing?" This is an indication of the fact 
that they really don't know because they are not that familiar with the process. They are 
looking to you for some positive feedback. It is good practice to provide them with some 
positive encouragement of a genuine nature if such is warranted. Such statements as: 
"I feel that we have done some really good work in this session" or "we have really had 
a productive session with a lot of good contributions" can help a great deal in sustaining 
their interest. At times the group may start to turn a little stale or hit a stumbling point. 
That is a good time to take a brief break or make the promise of one such as, "let's just 
finish this point and we'll take a break" or "this is a natural breaking point." In addition 
to or in lieu of the break it is sometimes desireable to have the group review the Matrix 
of Educational Effects to help them resolve some issues; this is especially so for 
specifying activities and resources. 
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WHAT TASKS ARE DONE IN EACH SESSION: 

A list of workshop tasks taken from Table 4.1 is given on the next page along with an 
indication of the session in which they may be first dealt with. Depending upon the 
. elapsed time between sessions 1 and 2, the workshop participants may need a little 
refresher (not to mention the Facilitators)) of what was done in the 1st session. 
Hopefully, the Organizational Contact will have gotten all the materials from the 1st 
session typed up and distributed beforehand so that all can readily review the results as 
a group before taking the next steps. I regard the review of stakeholder interview results 
as a "fresh brain" task which should be taken up as early as possible and finished with 
as little disruption as possible (viz., intervening tasks or events). Consequently we 
usually take up the interview results very soon after the resumption of the 2nd session 
(e.g., 15-20 minutes after beginning). This then allows a full afternoon or morning to 
complete them and some additional time later if it is needed. 

Once the interview results have been completed the modeling can be resumed. This is 
where a lot of memory refreshing needs to be done. Picking up with the Barriers is 
usually an ideal place to start (if you got that far) for it forces the participants to review 
the Functional com r * ..ent and Logic model. If you didnl get this far then you may want 
to revisit the Educai.onal Effects and specify their indicators if they havent been done yet 
(sometimes it is easier to save specifying these effects indicators for later and get right 
into the Functional component). The important point to be made is that it is nice to have 
a point of resumption of the modeling that forces the participants to review and think 
about what they did in the 1st session. 

Mortvedt (1990; 1991) has developed an ingenious set of worksheets which not only 
facilitate the transition from one session to the next but actually facilitate all of the barrier 
and spinoff identification work. Each work group member receives a set of worksheets 
wherein each sheet contains a representation of the Program Logic Model with a shading 
of the main event being focused on. Beneath each main event are given the specific 
activities and indicators with spaces to write in the specific barriers, barrier reductions, 
spinoffs and resources. Once completed, such an approach of presenting successively 
more details of the modeling, can also be used to familiarize others with the results. 
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List of Workshop Tasks 



Task Number/Session Number Title 

1. /One Introduction to Concepts and Procedures With an 

Example. 

2. /One Identification and Discussion of General Categories of 

Stakeholders. 

3. /One Formulation of Program Logic Model 

a. Educational Effects and Indicators 

b. Consequent Events, Effects and Indicators 

(indicators may be optional for the latter two) 

c. Antecedent Events 

4. /One Specification of Program Functional and Indicator 

Components. 

5. /One Development/Adaptation of Specific Stakeholder 

Questions. 

6. /One Identification of Individual Stakeholders to be 

Interviewed. 

7. /One Development of Stakeholder Contact Letters and 

Interview Procedures. 

8. /One Development of Plan for: Conduct, Transcription and 

Analysis of Interviews; Document Review and Analysis 

9. /One &/or Two Identification of Resources. 

10. /One &/or Two Identification of Barriers and Barrier Reductions. 
1 1 ./One &/or Two Identification of Spinoffs. 

12. /Two Review and Summary of Interview Results. 

13. /Two Develop Conclusions, Recommendations and Next 

Steps; 

Conduct Administrative Briefing. 



Optional (Later Session Tasks) 

14. Model Verification Steps. 

15. Individual Plans of Work Keyed to Events and Activities. 
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WHAT GUIDANCE IS GIVEN TO THE ORGANIZATIONAL CONTACT PERSON: 



In addition to the facilities requirements identified in the Workshop Format section, the 
Organizational Contact may need some guidance on matters related to the interviews 
and their analysis. They are: 

o The person or persons who do the interviewing must be independent of 
the topical area for that particular organization (often a graduate student 
or even super-Secretary can do the job nicely). 

o Phone interview equipment - a device that operates directly off the phone 
jacks can be purchased very inexpensively, if not already available, and 
works much better than one that attaches to the speaker. 

o Some materials giving directions to the interviewer concerning questions 
that may arise in the interviews may have to be provided--see the case 
reports referenced in the text or references for examples of these. 

o Opening Statement - this needs to be provided and is absolutely critical 
in starting the interview while not providing answers to the questions that 
follow. Again see some of the case reports. 

o A lot of time and effort can be saved by obtaining the names, addresses 
and phone numbers of stakeholders to be interviewed (if available) at 
some point (e.g., during a break) when the workgroup is still together 
instead of after they disperse. If they dont have it on hand, you can 
have them phone in to your secretary. 

o Case reports should be studied carefully for analysis methods, 
procedures and formats. 

o Responsibility for who is going to type up the interim materials (those 
from session 1) and who is going to prepare and have typed up the case 
report should be designated early on-preferably before the 1st session 
begins. 
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WHAT MATERIALS ARE USED AS A HANDOUT AND REFERENCE FOR 
WORKSHOP PARTICIPANTS: 

The following pages are used by the facilitator(s) in making their introduction to the 
program design process, One of the case reports from Appendices E-l (of Mayeske, 
1991 , Vol 2) is used as a handout to serve as the actual example. 
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HANDOUTS A 
FACILITATORS N OTES FOR EACH HANDOUT PAGE 

FACILITATORS NOTES FOR PROGRAM DESIGN PROCEDURES 

(PAGE 1 OF HANDOUT) 

The design team uses stakeholder* perceptions.what is known about the actual or 
intended program (from research, conduct of the program or related programs in the 
past, evaluations and other documents, etc.) combined with their prior experiences 
and collective expertise to develop the design of a program and identify actions to be 
taken on its behalf. The process that the group engages in uses extensive modeling 
exercises to develop a framework for what the program is or might be. As a result the 
team 

members usually arrive at a consensus and develop a sense of team membership. 

Futuristic perspectives about the pu qram or concerning trends that might affect the 
program can enter into the discussion in one or more of three ways: (1) through the 
viewpoints of the stakeholders; (2) through the experiences and views of the design 
team members; and, (3) via the organization's vision, mission and functions. If the 
organization has gone through some recent futuristic work (e.g. scanning, visioning, . 
etc.) then some brief time should be allowed for the group to review this work and 
discuss its significance for their current task. Hopefully, a half hour or less would 
suffice. 



FACILITATORS NOTES FOR PROTOTYPE WORKSHOP AGENDA 

(PAGE 2 OF HANDOUT) 

Discuss these steps briefly with the design teaam so that they have an idea of where 
they are headed. See Chapter 4 for an overview & Chapters 5-8 for more details... 



FACILITATORS NOTES-FOR-GENESIS OF PROGRAM DESIGN 

(PAGE 3 OF HANDOUT) 

The important points to be made are that these procedures have been especially 
adapted to Extension kinds of programming and as such they have been and are 
continuing to be used successfully in a wide variety of settings. Emphasis should be 
given to the topics dealt with. See Chapter 4 for more details. 



* A stakeholder is an individual (or group) who has a special interest in or influence 
over the program/topic and who can provide information that will be useful to the 
workgroup for the design, development, implementation and evaluation of the 
program. 
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BENEFITS FROM THE PROGRAM DESIGN PROCESS 
(PAGE 4 OF HANDOUT) 

It is important to emphasize that the process gives staff the time and opportunity to 
get together and go through a disciplined series of steps which results in the 
framework for a program -called the program design. The incorporation of 
stakeholder viewpoints contributes to and strengthens the deliberations of the group 
as well as making for a better design. The resultant design: 

•provides a basis for the plausibility of the program and its results (viz. how 
convincing is it that this program will lead to these expected results?) 

•serves as a tool for communication to stakeholders about different aspects of 
the program including needed resources, new staff.etc. 

•provides a blueprint for future action including program development and 
implementation 

•allows staff to sharpen their program planning and evaluation skills 



FACILITATORS NOTES FOR THE GENERIC PROGRAM LOGIC MODELS 
(PAGES 5,6,7 & 8 OF HANDOUT) 

Emphasize that the program logic model is the roadmap or organizing structure for 
all that follows. The logic model is comprised of a series of main events -or big 
happenings-major categories of important occurrences that are related in a sequential 
and causal manner. For any one event to occur, all those events preceding it must 
have occurred first. The IF-THEN sequencing emphasizes this causal relationship. For 
example, if MAIN EVENT #1 has occurred then MAIN EVENT #2 can occur, if 2 then 
3, if 3 then N, if N then the Educational Effects can occur and if these occur, then 
there can be Consequences. However, if for example MAIN EVENT #3 does not occur 
then nothing that follows it can occur. We'll see an example of -such a model in a 
moment. 

We start building this program logic model by identifying what the Educational Effects 
of the program are or will be(#5): Once these have been completed (#6) we identify 
Indicators of their occurrence(the dashed box) and then go on to identify other aspects 
of the model: Consequences and their Indicators and then cycle back to identify the 
events that must precede them.This takes us into the program itself where we identify 
the activities that make up each Main Event (the blank boxes), the resources to 
carry them out (the starred boxes) and the indicators for the occurrence of these 
activities (the dashed boxes). 

Once these have been developed we go on to (#7) identify things that can go wrong 
in carrying out the program-called barriers-and things that the staff might do about 
them if they do occur-called barrier reductions. We also identify things that can 
perturb the Educational Effects leading to the Consequences over which the staff may 
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have little or no influence-called intervening events. Finally, we identify unplanned 
occurrences that result from the program being structured and carried out in a 
particular manner-they are called spinoffs (#8). Examples of spinoffs are that Agents 
and Volunteers serve as role models for youth in the 4-H program or that Volunteers 
expenence psychic income and time management problems by virtue of their beina a 
Volunteer. a 



FACILITATOR NOTES FOR THE MATRIX OF EDUCATIONAL EFFECTS 
WORKSHEET (PAGE 9 OF HANDOUT) 

Note that the KASAB's come out of the Bennet hierarchy- spell each one out For 
each target audience we identify entries that reflect what changes clientele (or target 
audiences) will experience as a result of their participation in the program-in terms of 
these attnbutes. 



NATURE OF ENTRIES FOR THE MATRIX OF EDUCATIONAL EFFECTS 
(PAGE 10 OF HANDOUT) 

2o^o«l h With thG gr0Up the nature of the entries for each of t he KASAB's in turn 
EMPHASIZE that there does not have to be an entry in each cell for each taraet 
audience- there can be blanks and that is OK. 



FACILITATORS NOTES FOR THE SAMPLE STAKEHOLDER INTERVIEW 
QUESTIONS FOR PROGRAM XYZ (PAGE 11 OF HANDOUT) 

Review the questions & their purposes with the design team reminding them that they 
can be adapted or that they may be rejected as wholly inappropriate & that they can 
develop ones that are better suited to their circumstances. 



FACILITATORS NOTES FOR STAKEHOLDER CONTACT LETTERS & 
POINTERS (PAGES .12^.13 & 14 OF HANDOUT) 

2°w?f2! d ? UCh . I™ ° n theSe other than t0 review them brief, y and make the points 
that: (1) the Retail Trade Program had a considerable amount of prior experience 
wheareas the Water Quality effort had little; and, (2) emphasize strongly the footnote 
about not identifying the questions or providing answers to them. The sample letters 
will be useful in helping the team to draft their contact letter. 

Pages 13 & 14 should be reviewed with the team. However, the points are fairly self- 
explanatory. 7 
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CATEGORIZATION AND PRIORITIZATION OF STAKEHOLDERS 
AND STAKEHOLDER CATEGORIES 

• NO FIXED NUMBER TO INTERVIEW 

- USUALLY GET TOO MANY NOMINEES 
AND MUST REDUCE NUMBER 

• TRANSCRIPTION CAN BECOME COSTLY AND ANALYSIS TIME 

CONSUMING AFTER ABOUT 30 INDIVIDUAL INTERVIEWS 
OR 4 FOCUS GROUPS 

• THE CRITERION OF WHO HAS OR MIGHT HAVE A DIRECT 
EFFECT ON THE PROGRAM-TO-BE HAS TO BE USED 
REPEATEDLY 

• NEED TO DIFFERENTIATE BETWEEN THOSE WHO MAY OR MAY 

NOT BE KNOWLEDGEABLE AND DECIDE UNDER WHAT 
CONDITIONS SUCH INTERVIEWS WOULD BE USEFUL 
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SOME POINTS ON INTERVIEWING STAKEHOLDERS 



PAY SOME ATTENTION TO THE INITIAL LETTER RE 
STATUS OF SIGNATORY 
STUDY PURPOSES AND DETAIL 
ANONYMITY AND NON-ATTRIBUTION TO INDIVIDUALS 

CONTACT LETTER AND OPENING STATEMENT BY INTERVIEWER 
ARE CRITICAL IN ENCOURAGING PARTICIPATION WITHOUT 
ANSWERING THE QUESTIONS TO BE ASKED 

- MAY DO INDIVIDUALLY OR IN GROUP SETTINGS 
MAY DO FACE-TO-FACE, PHONE OR COMBINATION 
DONT INTERVIEW PEOPLE YOU KNOW WELL-USE A THIRD 
PARTY 

TAPE RECORD TOTAL INTERVIEW, TRANSCRIBE VERBATIM AND 
ANALYZE TRANSCRIPTIONS 

DEALING WITH PERCEPTUAL DATA FROM A JUDGMENT SAMPLE- 
NOT INTENDED TO BE STATISTICAL 

PURPOSE OF INTERVIEW IS "NOT" TO EDUCATE OR RE- 
EDUCATE THE STAKEHOLDER 



Appendix B 



Frequent Questions and Answers 



Q: How can you expect these models to have any validity when we know that people don't 
really behave that way? 

A: Real life is truly dynamic and non-linear. By forcing it to stand-still and be linear we are really 
creating an artificial condition that makes it easier to think about, talk about and 
communicate to others. 

Q: How can you talk of verifying models if people dont really behave that way? 

A: By verifying the model we mean that we try to determine the extent to which there is 
consensus among program providers that the model reflects or encompasses their 
experience in carrying out the prospective program, not that they will actually carry it out that 
way. The models are conceptual not physical; they are meant to serve as a guide to 
thought, action and communication and not as a description of actual thought and action. 

Q: Just how detailed can and should this modeling be? 

A: The modeling can be as detailed as one needs it to be. It can be carried to the level of 
Gant and Pert charts that will paper the walls of many rooms, if so desired. However, such 
a level of detail may be overwhelming and defeat the purposes of using it as a guide to 
thought, action and communication. Then too, the time and resources required to develop 
this level of detail may be more than anyone wants to invest. 

Q: Why aren't feedback loops allowed in the program logic model? 

A: We dorft use feedback loops because the arrows are supposed to indicate causa! 

relationships and we havenl allowed for "reverse" causation. We have on occasion drawn 
a single loop feeding back from the accomplishment of the goats/objectives to an earlier 
point, usually something related to an assessment or reassessment of the problem, issue or 
need because that would be a logical next step in the causal sequence ( see especially 
Chapter 13 ). In addition, a kind of feedback loop occurs when one reaches out to 
stakeholders &/or prospective clientele & experts to obtain their guidance on various 
concerns. 

Q: Is it necessary to do stakeholder interviews and how critical are they to the process? 

A: Both stakeholder interviews and program modeling can be done alone or in combination. 
Stakeholder interviews usually enlighten and enrich the program modeling process and 
provide some indication of how and what people think about the topic or prospective 
program. Stakeholder interviews can be useful in their own right as indicators of what critical 
questions and thoughts others might have about a topic. At times stakeholder interviews 
may not be done for part (or even all) of an identified group because the "political climate" 
could lead to a misinterpretation of the intent of the interviews or their results could be 
misconstrued and lead to undesirable consequences. Or, a program or project proposal is 
being prepared and time or resources do not allow for them. Alternatively, program 
modeling may not be done because the schedule for an evaluation of a program or the 
sparse level of funding precludes such, These comments should not be construed by those 
who have a genuine "fear of talking to stakeholders that they should not do so. The 
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techniques used together are far more powerful than either one alone. 

Q: Shouldn't we have a third party "expert" perform an objective and independent content 
analysis of the interview results so as to avoid any possible biases on the part of the 
workgroup? 

A: This question assumes that the analyses could be done independently by a third party 
expert in such a way that the results would be meaningful to the workgroup members. 
Preliminary analyses can be done by such a person and presented to the workgroup 
members for their consideration. However, the members themselves are the content 
experts. For the results to be meaningful and hence useful the workgroup members must 
review, interpret and categorize them or make recommendations as to how they might be 
interpreted and categorized. Pursuant to such guidance further analyses can be done by the 
third party "expert" (see Krippendorf (1980) for a variety of techniques that can be used and 
Mortvedt, 1990; 1991; for analysis examples). 

Q: Can this program design process be used for other types of programs, like research and 
development, as well as for educational programs? 

A: This approach employs some elementary principles that have wide applicability. What we 
have called the Matrix of Educational Effects can be regarded in more general terms as any 
Effects matrix. For such a matrix we could pose more general questions of: 



Answers to these questions provide one with the rudiments of a program which can then be elaborated 
on further using our concepts (logic model, etc.) or those of others. 



Q: Would one person want to use these techniques to develop their own plan of work? 

A: Not necessarily. They might require more time and more detail than is desirable. Then too, * 
it's fairly easy to reach a consensus with oneself albeit the conversation may not be as rich 
nor the focus as broad. However, it is difficult to imagine that a quick run through of most of 
the concepts wouldn't enrich the process and help produce a better plan. 

Q: Arenl prospective clientele of the program stakeholders too? 

A: Sometimes yes and sometimes no. It is often the case that the clientele or prospective 
clientele of a program are not well informed about the topic. Exceptions occur when former 
dientele of a program continue a relationship with Extension over a period of time. We call 
them cilentele-cooperators. They may very well be stakeholders and an excellent source 
of guidance. 

Q: How do the criteria and procedures set forth here relate to the criteria of quality and 
excellence as set forth by Mueller (1991) and Smith (1991)? 

A: There are many commonalities between these sets of criteria and what has been explicated 
in this manual. However, more procedural detail is provided herein. 



1. 
2. 
3. 
4. 



What will be different? 
In what ways will it or they be different? 
What happens as a result of (2)? 
What do we do to bring about (2)? 
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Q: A lot of these criteria have the appearance of a "recipe" in that if one follows them closely 
enough they will end up with a "tasty" result. How do they contrast with staff commitment 
and zeal? 

A: There is no substitute for staff commitment and zeal. They are undoubtedly the most 
important ingredients for programs to be successful. Sometimes they are the ingredients 
that make a program successful in spite of a set of requirements. However, when activities 
are to be orchestrated among a set of actors it seems to make sense to have them develop 
their thinking and reach some consensus on what it is they are about. 

Q: Are there any shortcuts to Program Design (PD) &/or Evaluability Assessment (EA)? 

A: A shortcut to EA that might be used is to adapt the stakeholder questions from Table 7.1 so that 
they can be used to interview program staff. On the basis of the interview results ( viz. the presence or 
absence of consensus or convergent/divergent responses) one can decide whether or not a "program" 
is in operation. If one is, then some modeling might be done, if one isn't, then effort may be needed to 
develop one. The extent to which viewpoints are diverge might serve as an indicator of the nature & 
extent of design work needed. [Alternatively, a checklist might be developed to be used in lieu of the 
interviews provided that "distorted responses" would not be a problem.] In short, if there is no 
consensus among staff as to what they are about, why go further? 

A shortcut to PD might be developed by using the illustrative model in Chapter 9 as a starting 
point. The model could be adapted in a sequential manner starting with the Program Logic Model, then 
the functional and indicator components, then barriers/reductions, intervening events and spin-offs, etc.. 
The fewer the adaptations made, the more directly useful the model would be. If nothing else, the 
model would serve to provoke the design group's thinking even though they might totally reject the 
model in its present form. The process might be greatly facilitated by some checklist or template 
approach. 

Q: How does the Life Cycle approach work if an organization has many such programs & teams? 

A: In Extension it is anticipated that an organization would have only a few programs that would 
warrant a sufficiently high priority to be handled in the Life-Cycle format with oversight from a Life 
Cycle Guidance Team (LCGT). If many such programs were in operation, then strong organizational 
tendencies would develop to have them all function within a common framework with the same reporting 
system and administrative structure with the LCGT functions performed by an executive. or 
administrative council. This "forced fit" of all programs into a common structure often weakens individual 
programs in fundamental ways. For example, the reporting system often becomes a numbers "game" 
with reporting schedules out of phase with the program cycle and with little or no feedback provided. 
The most negative problems from the most troublesome programs often dominate the agenda of the 
council so that little oversight and recognition is given to those doing well, etc.. In view of such 
considerations, the Life Cycle format might best be restricted to a few key programs. 



Q: Is it possible to develop programs that will have a "life of their own"? 

A: In their initial stages, programs can be devised so that the organization can "step back" and let the 
program operate almost, if not totally on its own. We noted such examples as the Master Gardner 
program in Chapter 13. If the design team keeps some of the following considerations in mind, then 
"stepping back" might becomeeasier: (1) involve volunteers early on in the program, perhaps recruiting 
them from the first wave of alumni, if necessary; (2) pursue a master/mentor model in which the more 
experienced volunteers can gradually take over supervisory and administrative functions; and, (3) 
implement the program in sites that have an infrastructure in place that can carry the program on its 
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own. Undoubtedly, the reader can think of yet other considerations. 



Q: One sometimes encounters a great deal of resistance to a formalized approach to programming such 
as is Deing advocated in the Life Cycle format. Why is this so & what might i do about it? 

A: Resistance has been found to such an approach in units that involve few staff & in which the staff is 
accustomed to functioning in an informal manner. In larger units the process gives staff who see each 
other very little a chance to plan jointly with the product providing a form of guidance for all. Little may 
come from any of these efforts if the administration does not provide appropriate incentives & support. 

Q: How does this approach relate to each of the following? 

o The Program Theoretic Approach in Evaluation 

tha orcgram theoretic movement appears to be receiving increased emphasis in the 
evaluation hrerature (Chen, 1990; Bickman, 1990; 1987; Patton, 1990) even though the 
notion can be traced back to the work of Suchman (1967) or even earlier if one includes 
chain«of-events, hypothesis trail, path model and flow-chart concepts. This approach grew 
out of that same tradition. However, we opted for our own terminology because it is closer 
to the approach devised by Wholey (1979) as adapted by Smith 
(1989a). 

o Strategic and Long-Range Planning 

As outlined by a number of authors (Armstrong, 1985; Bryson, 1989; Coates, 1986; 
Morrison, et. al., 1984; Simerly, 1987) this approach can pick up about where strategic and 
long-range planning leave off. That is, once a sense of the organization and where its going 
has been established this approach becomes one way of planning specific, discrete 
programs. 

o Project Management Procedures 

Many of the principles used in this approach are also used in a variety of approaches to 
project development and management (Kerzner, 1989; 1984; Kezsbom et. al., 1989; Muther, 
1988). 

o Issues Based Programming 

Extension has recently given great emphasis to the concept of "issues-based programming" 
(Albrecht and Deshler, 1990; Ules and Mustian, 1990; Brazzel and Sanderson, 1990; Taylor- 
Poweil and Upke, 1990). This approach readily lends itself to issues as well as needs and 
problems and especially so if they are interdisciplinary in nature. 



o The Paradigm Debate in Evaluation 

One of the things we have attempted to accomplish with this approach is to get evaluators 
and their techniques "out of the back of the bus and put them up with the driver where they 
are sorely needed". Since we remain a strong advocate of the use of multi-method research 
techniques for program planning and development (Brewer and Hunter, 1989) we have not 
totally escaped from this lofty debate (Guba and Lincoln, 1989; Patton, 1990). It suffices to 
say that all we ever really have are theories or hypotheses and there a variety of ways or 
methods that can be used to develop and refine them (Boone, 1985). 
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The Objectives Referenced Approach to Curriculum and Instructional Program Development 



One finds advocated frequently the approach in which learning objectives (and goals) are 
specified in concrete, numerical terms as to how many members of a given target audience 
will be served and how many will acquire specified levels of proficiency in certain skills 
(Tyler, 1971 ; Mager, 1972; 1975). These are usually developed on the basis of a logical or 
empirical analysis of the task(s) to be performed and may even guide the development of an 
instructional system (Gagne\ 1987). In the Program Design (PD) approach, a close 
counterpart to these numerically specified objectives can be obtained by specifying, in 
advance, numerical levels or targets for many of the indicators, especially for the educational 
effects indicators if they, themselves have been made specific enough. 
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Appendix C 



Sample Outline of a Case Report 



I. Overview/Summary 

II. Background and Conditions Supporting the Need and Purpose for Such an Effort 

ill. Present at ion of the New Program Logic Model as the Expected Program 

[ This model should be the organizing principle for the report & should be given major emphasis 
throughout.] 

IV. Identification of Work Group Participants and Organizational Status 

V. Discussion of the Topics dealt *vith and work completed in the sessions: 

Session I: Modeling; Stakeholder Identification and Questions 
Developed/Adapted 



Session II: Analysis and Summary of Interview Results including: 

Matrix of Summaries by Questions and Stakeholder Categories 
List of Thematic Observations from Interview Results 
Completion of Modeling 
Conclusions and Recommendations 
Next Steps and Lessons Learned 



VI. Appendices 

Program Logic Model, Functional and Indicator Components with Resources 

Expanded Program Logic Model with Barriers, Barrier Reductions, Intervening Events & Spin-offs 

Interview Results: number planned; number completed; reasons for non- 
completion; how conducted and by whom with what training and experience etc.; 
tabulations and analyses; etc. 
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Appendix D 
Definitions of Terms & Concepts 

Plausibility * Its Importance, and Other Concepts 

All Life Cycle Program Management (LCPM) efforts are intended to make programs more impactful 
once they are developed and delivered (viz., increase the likelihood that they achieve their intended 
goals and objectives). However, in the absence of impact information one must rely on some precursor 
of impact-an indicator or forecaster of impact, if you will. The concept used for such purposes is 
plausibility. Plausibility is a judgment about the likelihood that a program, either proposed or in 
different stages of development/imptementation/redirection, will achieve its intended goafs and 
objectives. Short of impact, it Is the superordinate concept in LCPM -the one for which all of the 
modeling and stakeholder involvement & analysis Is Intended to serve. I Even when impact is known & 
as a result programmatic changes are made, one again has to return to the notion of plausibility as a 
mediating concept.] LCPM specifies the Life Cycle Guidance Team (LCGT) as the deliberative body 
that should make these judgments. LCPM also attempts to specify criteria by which which such 
judgments can be made. 

Smith (1989a) has defined plausibility as "a judgment about the extent to which necessary arc! sufficient 
conditions exist for a program to succeed (viz., whether activities are of the right type and amount to 
bring about the desired change)" (p. 6). On this and the following pages are given definitions of many 
of the concepts used in the preceding chapters. 

Definitions of Program Life Cycle Concepts 

Barrier Analysis: analysis of the occurrence of similar barriers among pairs of main events in the 
expanded program logic model. 

Barrier Reductions: actions that program staff can initiate which might help surmount, overcome or 
avoid the barrier(s). 

Barriers: events, states or conditions that can perturb the causal relationships and that program staff 
might be able to influence. 

Concomitant Events: main events which occur neaiy simultaneously in the program logic model. 

Design Team: the persons assigned the task of conducting the exercises together, also called the task 
team, the work group or, at times, the group. 

Educational Experience Profile: a statement of the nature & extent of exposure to learning 
opportunities that a given target audience or set of audiences will receive in terms of the four 
interdependent facets of: resources; content; time; and, context, wherein the relative considerations 
given to the four are influenced by access to a research/knowledge base. 

Expanded Program Logic Model: the program logic model expanded to include barriers, barrier 
reductions, spinoffs and intervening events. 

Faciiitator(s): person(s) who assist(s) the design team to complete the different design tasks, 
stimulate/clarify their thinking, and reach a consensus on different concerns; may also perform a similar 
role for the development and transition management teams. 
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Feedback: forms of information provided to groups, units, individuals, etc. on which syntheses &/or 
judgments can be made about overall & relative performance with such syntheses/judgments being 
communicated back to those providing the information. 

Fidelity of focus: the extent to which the LCGT has retained its focus on the problem to be redressed 
or ameliorated in the course of carrying out its oversight and decision-making functions. 

Former Clientele: former participants, graduates or alumni of the program. 

Impact: the behavior/practice change effects of a program and/or their consequences. 

IrvDepth evaluation study: an intensive examination ( in terms of time and resources) of a 
programmatic effort/topic to estimate its actual &/or potential results or to assess the potential of new 
ventures. 

Indicator Component: sources of evidence that can be examined to ascertain whether or not the 
activities, effects or consequences have occurred or are occurring. 

Intervening Events: conditions, states or occurrences which can perturb the causal relationship^), are 
subsequent to the educational effects and are usually beyond the influence of the program staff. 

Life Cycle Program Evaluation (LCPE): an assessment of the accomplishments, impacts and value of 
a program in redressing or ameliorating a problem. 

Life Cycle Guidance Team (LCGT): a group whose task is to sustain the plausibility of a program as it 
moves through the different stages experienced in carrying out a program. 

Life Cycle Program Management (LCPM): a process in which programs are managed so as to sustain 
their plausibility within & between the distinctive phases of: problem finding; program design; program 
development; program implementation; program maintenance & improvement; and,program redirection. 

Linkage Expert (is*): experience from prior cycles that members of the LCGT bring to the deliberations 
of the team. 

Main Events: major categories of program activities, effects and consequences which form the 
program logic model. 

Matrix of Educational Effects: K(nowledge), A(titudes), S(Wlls), A(spirations) and Behavior/practice 
changes that clientele experience by virtue of their participation in the program (can vary in time of 
occurrence from immediate through intermediate to longer-term effects). Often called KASAB's for 
short. 

Model Verification: a test of the extent to which the model as developed by the workgroup can be 
used to describe the way other program providers can or do carry out the program. 

Mutual Clientele Effects: when the educational effects achieved with one set of clientele have effects 
on other clientele of the program; such changes can occur concomitantly, reciprocally or sequentially. 

Neetf : "the value judgment that some group has a problem that can be solved" (McKillip,1987). Can be 
organized into a hierarchy in which higher order needs are based upon having met lower order needs. 
Distinctions can be made as to types: normative; perceived; relative; maintenance; and, overmet. 
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Organizational Liaison/Contact Person: one who serves as the point of contact for the design team 
and the facilitator(s) and insures that interviews and other tasks (e.g., analysis, typing proceedings, 
scheduling meeting times and places, report preparation, etc.) are carried out. 

Plausibility: "a judgment about the extent to which necessary and sufficient conditions exist for a 
program to succeed" (Smith, 1989a). 

Problem Finding: the means by which problems are sought out & identified; can be of 2 types: general 
& targeted problem finding. 

Problem: "a set of ongoing perceptions held about a constantly changing gap between a desired & 
existing state" (Van Gundy, 1988b). 

Problution: a series of redefinitions or alternative definitions until ono redefinition of the problem is seen 
as a solution. 

Program Functional Component: the sets of activities that must be conducted and the resources 
needed to insure that the events in the logic model take place. 

Program Design: a theoretical framework for describing the effects and consequences of a program 
as they are related to its development and implementation. 

Program Providers: those persons who work with the clientele of a program. 

Program Modeling: the development of some representation of the program using schematics & flow 
diagrams. 

Program Logic Model: a set of causally and sequentially related main events that define the program 
and it's consequences and conform to an "if-then" relationship (viz., for any event to occur all thoso 
preceding it must have occurred first). 

Program Theory: a system of beliefs that are probabilistic in nature & which link an organized set of 
activities with supporting resources to intended results. 

Program Clientele or Participants: person(s) or group(s) who participate in the program. 

Program: a theory which Jinks an organized set of activities with supporting resources to intended 
results. 

Project: a specific type of program that has a specified deliverable within a given time period. 

Resources: the amount of staff time (professional, para-professional, volunteer, support, etc.) in FTE's, 
materials and other expenditures (e.g., travel, facilities etc.) needed to carry out the program. 

Spin-offs: unplanned effects of carrying out the program (can be positive or negative; known or 
unknown; and, anticipated or unanticipated, to those who provide the program). 

Stakeholder: an individual (or group) who has a special interest in or influence over the topical area of 
program-to-be and who can provide information that will be useful for the design, development, 
implementation and evaluation of the program. 

Target Groups(s) or Audience(s): intended recipients of the program; also called intended clientele. 
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Washington, D.C. and continues to do so after thirty some years. His first assignment 
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research on personnel career systems, occupational performance criteria and executive 
decision making. He transferred to the then U.S. Department of Health, Education and 
Welfare (HEW). Initially, he directed an Operations Research Group that performed 
further analyses of the Educational Opportunities Data ( the renowned Coleman study) 
which culminated in the publication of four monographs and numerous other papers. As 
a resutt of this work he received national and international recognition for his analyses of 
the socio-cultural determinants of achievement. Later this group was transferred to 
program evaluation where he was responsible for a number of large national evaluations 
of what were then Titles I and III of the Elementary and Secondary Education Act. 
Included among these studies were: A Longitudinal Study of Educational Practices; A 
Study of Compensatory Reading Programs; and, A Study of the Sustaining Effects of 
Compensatory Education on Basic Skills. 

Upon his return to USDA he began working on the evaluation of Extension programs 
starting with a nutrition education program for low-income families and moving on to such 
topics as: volunteerism; 4-H and Youth Development; Extension as a nation-wide system; 
and, State funding for Extension. While in HEW he had worked with Joe Wholey and 
brought to USDA some familiarity with Evaluability Assessment (EA). Since many of the 
concepts of EA seemed appropriate for Extension it seemed reasonable to try to adapt 
them from use in "top-down" organizations to the "bottoms-up" kind of programming done 
in Extension. Through a partnership arrangement with "Midge Smith" of the University of 
Maryland this was achieved culminating in the publication of her book "Evaluability 
Assessment: A Practical Approach". A network of persons throughout the U.S. and 
territories was established to work on carrying out EA's. In the course of conducting these 
it became increasingly clear that there was a great need for more attention and effort to 
be paid to program planning and design. Hence, many of the EA techniques were 
modified slightly and used for purposes of program (re)design. These efforts resulted in 
a monograph entitled "Program Design: An Evaluation Approach". Life Cycle Program 
Management was in part an outgrowth of these efforts. It was also an outgrowth of a 
need to have a framework that encompassed not only program beginnings but program 
endings as well, for Extension had an increasing need for a systematic way of getting rid 
of a "surplus of programmatic obligations". For this latter, a project was initiated which is 
in the course of developing and testing program ending techniques in a number of States. 
If early experiences are a good indicator, this approach will prove fruitful, is likely to be 
adopted on a large scale system-wide, and may work closely with strategic planning 
efforts. 
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